You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/19 02:30:46 UTC
[GitHub] [spark] sunchao opened a new pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
sunchao opened a new pull request #34659:
URL: https://github.com/apache/spark/pull/34659
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR adds support for complex types (e.g., list, map, array) for Spark's vectorized Parquet reader. In particular, this introduces the following changes:
1. Added a new class `ParquetColumnVector` which encapsulates all the necessary information needed when reading a Parquet column, including the `ParquetColumn` for the Parquet column, the repetition & definition levels (only allocated for a leaf-node of a complex type), as well as the reader for the column. In addition, it also contains logic for assembling nested columnar batches, via interpreting Parquet repetition & definition levels.
2. Changes are made in `VectorizedParquetRecordReader` to initialize a list of `ParquetColumnVector` for the columns read.
3. `VectorizedColumnReader` now also creates a reader for repetition column. Depending on whether maximum repetition level is 0, the batch read is now split into two code paths, e.g., `readBatch` versus `readBatchNested`.
4. Added logic to handle complex type in `VectorizedRleValuesReader`. For data types involving only struct or primitive types, it still goes with the old `readBatch` method which now also saves definition levels into a vector for later assembly. Otherwise, for data types involving array or map, a separate code path `readBatchNested` is introduced to handle repetition levels.
5. Modified `WritableColumnVector` to better support null structs. Currently it requires populating null entries to all child vectors when there is a null struct, however this will waste space and also doesn't work well with Parquet scan. This adds an extra field `structOffsets` which records the mapping from a row ID to the position of the row in the child vector, so that child vectors will only need to store real null elements.
This PR also introduced a new flag `spark.sql.parquet.enableNestedColumnVectorizedReader` which turns the feature on or off. By default it is on to facilitates all the Parquet related test coverage.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Whenever read schema containing complex types, at the moment Spark will fallback to the row-based reader in parquet-mr, which is much slower. As benchmark shows, by adding support into the vectorized reader, we can get ~15x on average speed up on reading struct fields, and ~1.5x when reading array of struct and map.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
With the PR Spark should now support reading complex types in its vectorized Parquet reader. A new config `spark.sql.parquet.enableNestedColumnVectorizedReader` is introduced to turn the feature on or off.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added new unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973736175
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973807807
**[Test build #145429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145429/testReport)** for PR 34659 at commit [`10c3a45`](https://github.com/apache/spark/commit/10c3a450af62b0028601c862005a566d623af253).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r769129914
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
+ for (ParquetColumn column : JavaConverters.seqAsJavaList(parquetColumn.children())) {
+ checkColumn(column);
+ }
+ }
- String[] colPath = paths.get(i);
- if (fileSchema.containsPath(colPath)) {
- ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);
- if (!fd.equals(columns.get(i))) {
+ /**
+ * Check whether a column from requested schema is missing from the file schema, or whether it
+ * conforms to the type of the file schema.
+ */
+ private void checkColumn(ParquetColumn column) throws IOException {
+ String[] path = JavaConverters.seqAsJavaList(column.path()).toArray(new String[0]);
+ if (containsPath(fileSchema, path)) {
+ if (column.isPrimitive()) {
+ ColumnDescriptor desc = column.descriptor().get();
+ ColumnDescriptor fd = fileSchema.getColumnDescription(desc.getPath());
+ if (!fd.equals(desc)) {
throw new UnsupportedOperationException("Schema evolution not supported.");
}
- missingColumns[i] = false;
} else {
- if (columns.get(i).getMaxDefinitionLevel() == 0) {
- // Column is missing in data but the required data is non-nullable. This file is invalid.
- throw new IOException("Required column is missing in data file. Col: " +
- Arrays.toString(colPath));
+ for (ParquetColumn childColumn : JavaConverters.seqAsJavaList(column.children())) {
Review comment:
Thanks, yea I can raise another PR to use list in `ParquetColumn`.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
+ for (ParquetColumn column : JavaConverters.seqAsJavaList(parquetColumn.children())) {
+ checkColumn(column);
+ }
+ }
- String[] colPath = paths.get(i);
- if (fileSchema.containsPath(colPath)) {
- ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);
- if (!fd.equals(columns.get(i))) {
+ /**
+ * Check whether a column from requested schema is missing from the file schema, or whether it
+ * conforms to the type of the file schema.
+ */
+ private void checkColumn(ParquetColumn column) throws IOException {
+ String[] path = JavaConverters.seqAsJavaList(column.path()).toArray(new String[0]);
+ if (containsPath(fileSchema, path)) {
+ if (column.isPrimitive()) {
+ ColumnDescriptor desc = column.descriptor().get();
+ ColumnDescriptor fd = fileSchema.getColumnDescription(desc.getPath());
+ if (!fd.equals(desc)) {
throw new UnsupportedOperationException("Schema evolution not supported.");
}
- missingColumns[i] = false;
} else {
- if (columns.get(i).getMaxDefinitionLevel() == 0) {
- // Column is missing in data but the required data is non-nullable. This file is invalid.
- throw new IOException("Required column is missing in data file. Col: " +
- Arrays.toString(colPath));
+ for (ParquetColumn childColumn : JavaConverters.seqAsJavaList(column.children())) {
Review comment:
Thanks, yea I can raise another PR to use list in `ParquetColumn` after this one is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998353535
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146411/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r757856767
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
+ if (v == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
+ } else if (!state.isRequired && v == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, v);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // the current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // this RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
Review comment:
should leftInBatch be updated here as well ?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
+ if (v == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
+ } else if (!state.isRequired && v == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, v);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // the current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // this RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // the current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // the range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ leftInBatch -= n;
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
Review comment:
lets move leftInBatch update here.
I am not sure why some of these are local variables vs member vars :-)
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
+ if (v == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
+ } else if (!state.isRequired && v == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, v);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
Review comment:
The original non nested code refers to currentCount as this.currentCount in certain places. Should we pick that style as well ?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -897,6 +897,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_VECTORIZED_READER_NESTED_COLUMN_ENABLED =
+ buildConf("spark.sql.parquet.enableNestedColumnVectorizedReader")
+ .doc("Enables vectorized Parquet decoding for nested columns (e.g., struct, list, map). " +
+ s"Note to enable this ${PARQUET_VECTORIZED_READER_ENABLED.key} also needs to be enabled.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
I am liking the confidence :-)
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
Review comment:
v -> value ?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
+ if (v == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
+ } else if (!state.isRequired && v == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, v);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // the current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // this RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // the current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // the range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ leftInBatch -= n;
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
Review comment:
Not sure, but should repLevels be immutable for the purpose of this function ?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
Review comment:
I am not too familiar with this code, but where do we assert that these are the only two encoding choices. There is no default arm in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984306042
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145831/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r826407787
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
Review comment:
Yes, this does compare by reference. `parquetColumn` in `VectorizedParquetRecordReader` is created in its constructor, and then later on it is used in both constructing the `missingColumns` field and `columnVectors` field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r834918105
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel < maxElementDefinitionLevel) {
+ // This means the collection is null, and it is not an array element. In this case, we
+ // should increase offset to skip it when returning an array starting from the offset.
+ //
+ // For instance, considering an array of strings with 2 elements like the following:
+ // null, [a, b, c]
+ // the child array (which is of String type) in this case will be:
+ // null: 1 0 0 0
+ // length: 0 1 1 1
+ // offset: 0 0 1 2
+ // and the array itself will be:
+ // null: 1 0
+ // length: 0 3
+ // offset: 0 1
+ //
+ // It's important that for the second element `[a, b, c]`, the offset starts from 1
Review comment:
```suggestion
// It's important that for the second element `[a, b, c]`, the offset in the array (not the elements) starts from 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1084014921
lgtm. I may merge this in next few days if no more comments. cc @cloud-fan @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840992415
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -714,15 +718,47 @@ public WritableColumnVector arrayData() {
public WritableColumnVector getChild(int ordinal) { return childColumns[ordinal]; }
/**
- * Returns the elements appended.
+ * Returns the number of child vectors.
+ */
+ public int getNumChildren() {
+ return childColumns.length;
+ }
+
+ /**
+ * Returns the elements appended. This is useful
*/
public final int getElementsAppended() { return elementsAppended; }
+ /**
+ * Increment number of elements appended by 'num'.
+ *
+ * This is useful when one wants to use the 'putXXX' API to add new elements to the vector, but
+ * still want to keep count of how many elements have been added (since the 'putXXX' APIs don't
+ * increment count).
+ */
+ public final void addElementsAppended(int num) {
+ elementsAppended += num;
+ }
+
/**
* Marks this column as being constant.
*/
public final void setIsConstant() { isConstant = true; }
+ /**
+ * Marks this column only contains null values.
+ */
+ public final void setAllNull() {
Review comment:
OK, let's ignore this for the time being
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840639771
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -187,6 +188,30 @@ object ParquetUtils {
}
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ schema.forall(f => isBatchReadSupported(sqlConf, f.dataType))
+
+ def isBatchReadSupported(sqlConf: SQLConf, dt: DataType): Boolean = dt match {
+ case _: AtomicType =>
+ true
+ case at: ArrayType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, at.elementType)
Review comment:
nit: should be 2 spaces here
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -187,6 +188,30 @@ object ParquetUtils {
}
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ schema.forall(f => isBatchReadSupported(sqlConf, f.dataType))
+
+ def isBatchReadSupported(sqlConf: SQLConf, dt: DataType): Boolean = dt match {
+ case _: AtomicType =>
+ true
+ case at: ArrayType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, at.elementType)
+ case mt: MapType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, mt.keyType) &&
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840672386
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +215,379 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
}
}
- state.advanceOffsetAndRowId(offset, rowId);
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ /**
+ * Reads a batch of repetition levels, definition levels and values into 'repLevels',
+ * 'defLevels' and 'values' respectively. The definition levels and values are read via
+ * 'defLevelsReader' and 'valueReader' respectively.
+ * <p>
+ * The related states such as row index, offset, number of rows left in the batch and page,
+ * are tracked by 'state'. The type-specific 'updater' is used to update or skip values.
+ */
+ public void readBatchRepeated(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchRepeatedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ /**
+ * Reads a batch of repetition levels, definition levels and integer values into 'repLevels',
+ * 'defLevels', 'values' and 'nulls' respectively. The definition levels and values are read via
+ * 'defLevelsReader' and 'valueReader' respectively.
+ * <p>
+ * The 'values' vector is used to hold non-null values, while 'nulls' vector is used to hold
+ * null values.
+ * <p>
+ * The related states such as row index, offset, number of rows left in the batch and page,
+ * are tracked by 'state'.
+ * <p>
+ * Unlike 'readBatchRepeated', this is used to decode dictionary indices in dictionary encoding.
+ */
+ public void readIntegersRepeated(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchRepeatedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchRepeatedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // Values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // The current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // This RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // Need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // The current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // The range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // Skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // Read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
+ currentCount -= n;
+ leftInBatch -= n;
+ leftInPage -= n;
+ }
+ }
+ } else {
+ // Not a top-level row: just read all the repetition levels in the block if the row
+ // should be included according to row indexes, else skip the rows.
+ if (!state.shouldSkip) {
+ repLevels.appendInts(valuesLeftInBlock, currentValue);
+ }
+ state.numBatchedDefLevels += valuesLeftInBlock;
+ leftInPage -= valuesLeftInBlock;
+ currentCount -= valuesLeftInBlock;
+ }
+ break;
+ case PACKED:
+ int i = 0;
+
+ for (; i < valuesLeftInBlock; i++) {
+ int currentValue = currentBuffer[currentBufferIdx + i];
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ break;
+ } else if (rowId < rangeStart) {
+ // This is a top-level row, therefore check if we should skip it with row indexes
+ // the row is before the current range, skip it
+ defLevelProcessor.skipValues(1);
+ } else if (rowId > rangeEnd) {
+ // The row is after the current range, move to the next range and compare again
+ state.nextRange();
+ break;
+ } else {
+ // The row is in the current range, decrement the row counter and read it
+ leftInBatch--;
+ repLevels.appendInt(0);
+ defLevelProcessor.readValues(1);
+ }
+ rowId++;
+ } else {
+ if (!state.shouldSkip) {
+ repLevels.appendInt(currentValue);
+ }
+ state.numBatchedDefLevels += 1;
+ }
+ }
+
+ leftInPage -= i;
+ currentCount -= i;
+ currentBufferIdx += i;
+ break;
+ }
+ }
+
+ // Process all the batched def levels
+ defLevelProcessor.finish();
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ private static class DefLevelProcessor {
+ private final VectorizedRleValuesReader reader;
+ private final ParquetReadState state;
+ private final WritableColumnVector defLevels;
+ private final WritableColumnVector values;
+ private final WritableColumnVector nulls;
+ private final boolean valuesReused;
+ private final VectorizedValuesReader valueReader;
+ private final ParquetVectorUpdater updater;
+
+ DefLevelProcessor(
+ VectorizedRleValuesReader reader,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ this.reader = reader;
+ this.state = state;
+ this.defLevels = defLevels;
+ this.values = values;
+ this.nulls = nulls;
+ this.valuesReused = valuesReused;
+ this.valueReader = valueReader;
+ this.updater = updater;
+ }
+
+ void readValues(int n) {
+ if (!state.shouldSkip) {
+ state.numBatchedDefLevels += n;
+ } else {
+ reader.skipValues(state.numBatchedDefLevels, state, valueReader, updater);
+ state.numBatchedDefLevels = n;
+ state.shouldSkip = false;
+ }
+ }
+
+ void skipValues(int n) {
+ if (state.shouldSkip) {
+ state.numBatchedDefLevels += n;
+ } else {
+ reader.readValues(state.numBatchedDefLevels, state, defLevels, values, nulls, valuesReused,
+ valueReader, updater);
+ state.numBatchedDefLevels = n;
+ state.shouldSkip = true;
+ }
+ }
+
+ void finish() {
+ if (state.numBatchedDefLevels > 0) {
+ if (state.shouldSkip) {
+ reader.skipValues(state.numBatchedDefLevels, state, valueReader, updater);
+ } else {
+ reader.readValues(state.numBatchedDefLevels, state, defLevels, values, nulls,
+ valuesReused, valueReader, updater);
+ }
+ state.numBatchedDefLevels = 0;
+ }
+ }
+ }
+
+ /**
+ * Read the next 'total' values (either null or non-null) from this definition level reader and
+ * 'valueReader'. The definition levels are read into 'defLevels'. If a value is not
+ * null, it is appended to 'values'. Otherwise, a null bit will be set in 'nulls'.
+ *
+ * This is only used when reading repeated values.
+ */
+ private void readValues(
+ int total,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ defLevels.reserveAdditional(total);
+ values.reserveAdditional(total);
+ if (!valuesReused) {
+ // 'nulls' is a separate column vector so we'll have to reserve it separately
+ nulls.reserveAdditional(total);
+ }
+
+ int n = total;
+ int initialValueOffset = state.valueOffset;
+ while (n > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+ int num = Math.min(n, this.currentCount);
+ switch (mode) {
+ case RLE:
+ if (currentValue == state.maxDefinitionLevel) {
+ updater.readValues(num, state.valueOffset, values, valueReader);
+ state.valueOffset += num;
+ } else {
+ // Only add null if this represents a null element, but not the case when a
+ // collection is null or empty.
+ nulls.putNulls(state.valueOffset, num);
+ state.valueOffset += num;
+ }
+ defLevels.putInts(state.levelOffset, num, currentValue);
Review comment:
line 556 and 561 (`state.valueOffset += num;`) can move before line 563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840701302
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -714,15 +718,47 @@ public WritableColumnVector arrayData() {
public WritableColumnVector getChild(int ordinal) { return childColumns[ordinal]; }
/**
- * Returns the elements appended.
+ * Returns the number of child vectors.
+ */
+ public int getNumChildren() {
+ return childColumns.length;
+ }
+
+ /**
+ * Returns the elements appended. This is useful
*/
public final int getElementsAppended() { return elementsAppended; }
+ /**
+ * Increment number of elements appended by 'num'.
+ *
+ * This is useful when one wants to use the 'putXXX' API to add new elements to the vector, but
+ * still want to keep count of how many elements have been added (since the 'putXXX' APIs don't
+ * increment count).
+ */
+ public final void addElementsAppended(int num) {
+ elementsAppended += num;
+ }
+
/**
* Marks this column as being constant.
*/
public final void setIsConstant() { isConstant = true; }
+ /**
+ * Marks this column only contains null values.
+ */
+ public final void setAllNull() {
Review comment:
1. `column.numNulls()` may still 0 after we call `setAllNull()`
2. do `column.setAllNull()` and `column.putXXX`, then `column.isAllNull` still true.
The above two scenes look strange
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803331855
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -144,6 +145,30 @@ object ParquetUtils {
file.getName == ParquetFileWriter.PARQUET_METADATA_FILE
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ schema.forall(f => isBatchReadSupported(sqlConf, f.dataType))
+
+ def isBatchReadSupported(sqlConf: SQLConf, dt: DataType): Boolean = dt match {
Review comment:
This looks like visible for testing only. If you don't mind, could you make this `private` and rewrite the only usage in `ParquetFileFormatSuite` with `invokePrivate`?
```
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormatSuite.scala: assert(ParquetUtils.isBatchReadSupported(conf, dt) == expected)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803979569
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
Review comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r774275032
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
Thanks! let me try to reproduce the above scenario (it'd be great if you already have some code snippet to share) locally and see what's the causing the perf issue in `ParquetToSparkSchemaConverter#convertInternal`. Ideally we can skip the code path if the schema doesn't contain complex types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r774270256
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
This line has a performance impact on parquet files with very wide records.
Take, for example, with a parquet dataset with:
- 6000 bigint columns
- 25 files
- 5000 records per file (total of 125000 records)
Reading all the records in this parquet dataset with a single executor thread is 42% slower with this PR than with the baseline (commit f361ad8d2e on the master branch).
The main culprit is the n**2 issue [here](https://github.com/apache/spark/blob/f361ad8d2eb067ec86483a294771f1976f31a981/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L95) in ParquetToSparkSchemaConverter#convertInternal, which was not exercised anywhere except by unit tests until your PR (as far as I can tell).
By the way, even with a patch for the above n**2 issue, this PR is still 10% slower than the baseline when testing the wide row case. I haven't confirmed it, but I think this is due to the extra OnHeapColumnVector instances created in ParquetColumnVector, for which you already have a TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998352669
**[Test build #146411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146411/testReport)** for PR 34659 at commit [`aff8504`](https://github.com/apache/spark/commit/aff8504d4130095ee1ecfdab3f3d2d9110d40400).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984184680
Thanks @sadikovi and @agrawaldevesh ! I've addressed most of your comments, while some left to be answered. Please let me know what you think.
> On a more style-like note: could you make all of the comments to start with an uppercase letter? I find it easier to read them this way as they indicate the beginning of a paragraph.
Sure, it's done.
> am curious how often are nested fields selected in the resultSchema given nested column pruning in effect ?
This is orthogonal I think. With or without nested column pruning users may still want to define nested schemas and efficiently read data of them. From my experience I've seen struct type being commonly used in data schemas and this PR should help a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984306042
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145831/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r829408672
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
I have the follow-up to skip creating these column vectors and reading definition levels if unnecessary done [here](https://github.com/sunchao/spark/commit/f968791635db1d679638511fc4ff01d2b1eccff0). I can include it to the PR if it doesn't complicated the review :) cc @sadikovi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r779210735
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
Thanks, yes I'm aware of the other one: we should not need to allocate the vectors for definition & repetition levels when the schema is flat. I'm hoping to address this separately with another PR though - don't want to make this one too bloated :)
BTW @bersprockets : how can I reproduce the 10-15% performance penalty? I was using the above code snippet and got almost the same numbers on my machine with the latest fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1064813892
Let me take another look, it seems there are have been quite a few changes since 🙂 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760514700
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parent is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel == maxDefinitionLevel - 1) {
+ // the collection is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // collection is defined and non-empty: find out how many top element there is till the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i + 1);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
+ int maxRepetitionLevel = column.repetitionLevel();
+ int maxDefinitionLevel = column.definitionLevel();
+
+ vector.reserve(definitionLevels.getElementsAppended());
+
+ int rowId = 0;
+ int nonnullRowId = 0;
+ boolean hasRepetitionLevels = repetitionLevels.getElementsAppended() > 0;
+ for (int i = 0; i < definitionLevels.getElementsAppended(); i++) {
+ // if repetition level > maxRepetitionLevel, the value is a nested element (e.g., an array
+ // element in struct<array<int>>), and we should skip the definition level since it doesn't
+ // represent with the struct.
+ if (!hasRepetitionLevels || repetitionLevels.getInt(i) <= maxRepetitionLevel) {
+ if (definitionLevels.getInt(i) == maxDefinitionLevel - 1) {
+ // the struct is null
+ vector.putNull(rowId);
+ rowId++;
+ } else if (definitionLevels.getInt(i) >= maxDefinitionLevel) {
+ vector.putNotNull(rowId);
+ vector.putStruct(rowId, nonnullRowId);
+ rowId++;
+ nonnullRowId++;
+ }
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private static WritableColumnVector allocateLevelsVector(int capacity, MemoryMode memoryMode) {
+ switch (memoryMode) {
+ case ON_HEAP:
+ return new OnHeapColumnVector(capacity, DataTypes.IntegerType);
+ case OFF_HEAP:
+ return new OffHeapColumnVector(capacity, DataTypes.IntegerType);
+ default:
+ throw new IllegalArgumentException("Unknown memory mode: " + memoryMode);
+ }
+ }
+
+ private int getNextCollectionStart(int maxRepetitionLevel, int elementIndex) {
+ int idx = elementIndex + 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
+ break;
+ }
+ }
+ return idx;
+ }
+
+ private int getCollectionSize(int maxRepetitionLevel, int idx) {
+ int size = 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
+ break;
+ } else if (repetitionLevels.getInt(idx) <= maxRepetitionLevel + 1) {
+ // only count elements which belong to the current collection
+ // For instance, suppose we have the following Parquet schema:
+ //
+ // message schema { max rl max dl
+ // optional group col (LIST) { 0 1
+ // repeated group list { 1 2
+ // optional group element (LIST) { 1 3
+ // repeated group list { 2 4
+ // required int32 element; 2 4
+ // }
+ // }
+ // }
+ // }
+ // }
+ //
+ // For a list such as: [[[0, 1], [2, 3]], [[4, 5], [6, 7]]], the repetition & definition
+ // levels would be:
+ //
+ // repetition levels: [0, 2, 1, 2, 0, 2, 1, 2]
+ // definition levels: [2, 2, 2, 2, 2, 2, 2, 2]
+ //
+ // when calculating collection size for the outer array, we should only count repetition
+ // levels whose value is <= 1 (which is the max repetition level for the inner array)
Review comment:
You mean `<= 1` in the sentence? it is about the above example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984181954
**[Test build #145831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145831/testReport)** for PR 34659 at commit [`7b6e083`](https://github.com/apache/spark/commit/7b6e083833c807e44b76bc321d32e9c5857e648f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760635579
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
Review comment:
The `MODE` enum is private to this class and the `mode` is initialized in this class, so I guess it should be fine. This maps to the Parquet spec for RLE/BitPacked encoding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao edited a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1034226787
Could you review this? @cloud-fan @dongjoon-hyun @viirya @huaxingao ? It'd be great if this can catch Spark 3.3 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803985141
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java
##########
@@ -176,8 +176,7 @@ public void initBatch(
// Initialize the missing columns once.
if (colId == -1) {
OnHeapColumnVector missingCol = new OnHeapColumnVector(capacity, dt);
- missingCol.putNulls(0, capacity);
- missingCol.setIsConstant();
+ missingCol.setAllNull();
Review comment:
This change is actually not related. I think we can we have a separate small PR for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r834918618
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel < maxElementDefinitionLevel) {
+ // This means the collection is null, and it is not an array element. In this case, we
+ // should increase offset to skip it when returning an array starting from the offset.
+ //
+ // For instance, considering an array of strings with 2 elements like the following:
+ // null, [a, b, c]
+ // the child array (which is of String type) in this case will be:
+ // null: 1 0 0 0
+ // length: 0 1 1 1
+ // offset: 0 0 1 2
+ // and the array itself will be:
+ // null: 1 0
+ // length: 0 3
+ // offset: 0 1
+ //
+ // It's important that for the second element `[a, b, c]`, the offset starts from 1
+ // since otherwise we'd include the first null element from child array in the result.
+ offset += 1;
+ }
+ if (definitionLevel <= maxDefinitionLevel - 1) {
Review comment:
why this is not `else if`? Could two cases happen at the same time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839044063
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
Review comment:
This is used for decoding dictionary indices. Will add some comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839068217
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
Review comment:
This is because `child.definitionLevel()` already incremented when the child is optional, so we need to negate it here. Still considering the example above. For optional elements:
```
RL DL
message schema { 0 0
optional group _1 (LIST) { 0 1
repeated group list { 1 2
optional binary element (UTF8); 1 3
}
}
}
```
The goal is to decide whether a value is an array element or not. Here `maxElementDefinitionLevel` is 3 - 1 = 2, and anything < 2 is not an array element.
```
RL DL
message schema { 0 0
optional group _1 (LIST) { 0 1
repeated group list { 1 2
required binary element (UTF8); 1 2
}
}
}
```
Similarly for required element, `maxElementDefinitionLevel` is 2.
I can change the variable name `maxElementDefinitionLevel` to something else if it's confusing :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r838976941
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -214,24 +231,32 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
boolean needTransform = castLongToInt || isUnsignedInt32 || isUnsignedInt64;
column.setDictionary(new ParquetDictionary(dictionary, needTransform));
} else {
- updater.decodeDictionaryIds(readState.offset - startOffset, startOffset, column,
+ updater.decodeDictionaryIds(readState.valueOffset - startOffset, startOffset, column,
dictionaryIds, dictionary);
}
} else {
- if (column.hasDictionary() && readState.offset != 0) {
+ if (column.hasDictionary() && readState.valueOffset != 0) {
// This batch already has dictionary encoded values but this new page is not. The batch
// does not support a mix of dictionary and not so we will decode the dictionary.
- updater.decodeDictionaryIds(readState.offset, 0, column, dictionaryIds, dictionary);
+ updater.decodeDictionaryIds(readState.valueOffset, 0, column, dictionaryIds, dictionary);
}
column.setDictionary(null);
VectorizedValuesReader valuesReader = (VectorizedValuesReader) dataColumn;
- defColumn.readBatch(readState, column, valuesReader, updater);
+ if (readState.maxRepetitionLevel == 0) {
+ defColumn.readBatch(readState, column, definitionLevels, valuesReader, updater);
+ } else {
+ repColumn.readBatchNested(readState, repetitionLevels, defColumn, definitionLevels,
+ column, valuesReader, updater);
+ }
}
}
}
private int readPage() {
DataPage page = pageReader.readPage();
+ if (page == null) {
+ return -1;
+ }
Review comment:
Yes, I have some comments in line 182 above:
> we've read all the pages; this could happen when we're reading a repeated list and we don't know where the list will end until we've seen all the pages.
For primitive types, we know a) the exact number of total values to read, and 2) the total number of values for each page. Therefore we know which page is the last one.
However, a repeated list could span multiple pages in Parquet, and therefore we don't really know which page is the last, until the Parquet page returns null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999945284
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146490/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999789529
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50965/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r778514200
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
Updated the PR. I also re-run the above test which used to take 1.05m before the fix, and improved to 0.58m afterwards, which is very close to the number in master branch.
@bersprockets it'd be great if you can verify it too. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803324113
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileBasedDataSourceTest.scala
##########
@@ -52,7 +54,8 @@ private[sql] trait FileBasedDataSourceTest extends SQLTestUtils {
f(spark.read.format(dataSourceName).load(path.toString))
}
if (testVectorized) {
- withSQLConf(vectorizedReaderEnabledKey -> "true") {
+ withSQLConf(vectorizedReaderEnabledKey -> "true",
+ vectorizedReaderNestedEnabledKey -> "true") {
Review comment:
Could you revise the test framework to have a complete test coverage for both `true` and `false`, @sunchao ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973736175
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-974257146
Thanks @sadikovi! take your time on this one :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984204113
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760525243
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
Review comment:
Yes it's it allowed. This use pattern is pretty standard in Java (in fact IntelliJ will warn you if you put the generic type there).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r826425700
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -187,6 +188,30 @@ object ParquetUtils {
}
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ schema.forall(f => isBatchReadSupported(sqlConf, f.dataType))
+
+ def isBatchReadSupported(sqlConf: SQLConf, dt: DataType): Boolean = dt match {
+ case _: AtomicType =>
+ true
+ case at: ArrayType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, at.elementType)
+ case mt: MapType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, mt.keyType) &&
+ isBatchReadSupported(sqlConf, mt.valueType)
+ case st: StructType =>
Review comment:
I'm not sure if we even support reading & writing of `CalendarIntervalType` in Parquet at the moment - it seems we don't from looking at `ParquetWriteSupport.makeWriter`. IIUC it is also going to be deprecated by `YearMonthIntervalType` and `DayTimeIntervalType`, is that correct @MaxGekk ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r834907480
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel < maxElementDefinitionLevel) {
+ // This means the collection is null, and it is not an array element. In this case, we
+ // should increase offset to skip it when returning an array starting from the offset.
+ //
+ // For instance, considering an array of strings with 2 elements like the following:
+ // null, [a, b, c]
+ // the child array (which is of String type) in this case will be:
+ // null: 1 0 0 0
+ // length: 0 1 1 1
+ // offset: 0 0 1 2
+ // and the array itself will be:
+ // null: 1 0
+ // length: 0 3
+ // offset: 0 1
+ //
+ // It's important that for the second element `[a, b, c]`, the offset starts from 1
+ // since otherwise we'd include the first null element from child array in the result.
+ offset += 1;
+ }
+ if (definitionLevel <= maxDefinitionLevel - 1) {
+ // Collection is null or one of its optional parents is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // Collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // Collection is defined and non-empty: find out how many top elements are there until the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
+ int maxRepetitionLevel = column.repetitionLevel();
+ int maxDefinitionLevel = column.definitionLevel();
+
+ vector.reserve(definitionLevels.getElementsAppended());
+
+ int rowId = 0;
+ boolean hasRepetitionLevels = repetitionLevels.getElementsAppended() > 0;
+ for (int i = 0; i < definitionLevels.getElementsAppended(); i++) {
+ // If repetition level > maxRepetitionLevel, the value is a nested element (e.g., an array
+ // element in struct<array<int>>), and we should skip the definition level since it doesn't
+ // represent with the struct.
+ if (!hasRepetitionLevels || repetitionLevels.getInt(i) <= maxRepetitionLevel) {
+ if (definitionLevels.getInt(i) <= maxDefinitionLevel - 1) {
+ // Struct is null
+ vector.putNull(rowId);
+ rowId++;
+ } else if (definitionLevels.getInt(i) >= maxDefinitionLevel) {
+ vector.putNotNull(rowId);
+ rowId++;
+ }
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private static WritableColumnVector allocateLevelsVector(int capacity, MemoryMode memoryMode) {
+ switch (memoryMode) {
+ case ON_HEAP:
+ return new OnHeapColumnVector(capacity, DataTypes.IntegerType);
+ case OFF_HEAP:
+ return new OffHeapColumnVector(capacity, DataTypes.IntegerType);
+ default:
+ throw new IllegalArgumentException("Unknown memory mode: " + memoryMode);
+ }
+ }
+
+ /**
+ * For a collection (i.e., array or map) element at index 'idx', returns the starting index of
+ * the next collection after it.
+ *
+ * @param maxRepetitionLevel the maximum repetition level for the elements in this collection
+ * @param idx the index of this collection in the Parquet column
+ * @return the starting index of the next collection
+ */
+ private int getNextCollectionStart(int maxRepetitionLevel, int idx) {
+ idx += 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
Review comment:
Isn't `repetitionLevels.getInt(idx) < maxRepetitionLevel`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998368830
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50886/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999945284
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146490/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r774696727
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
Thanks again @bersprockets ! let me address the issue and update the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998353521
**[Test build #146411 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146411/testReport)** for PR 34659 at commit [`aff8504`](https://github.com/apache/spark/commit/aff8504d4130095ee1ecfdab3f3d2d9110d40400).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998353535
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146411/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839063209
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
Review comment:
For example, an optional array with optional element is represented by:
```
RL DL
message schema { 0 0
optional group _1 (LIST) { 0 1
repeated group list { 1 2
optional binary element (UTF8); 1 3
}
}
}
```
here `RL` and `DL` are the max repetition and definition level for the schema element at the corresponding positions. The `maxDefinitionLevel` here is the max definition level of the array itself, which is 1. Any array element should have definition level > 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1086486426
Thanks all. Merging to master/3.3. @sunchao Please remember to open a follow-up PR to disable the config and update related tests before 3.3 release. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803968016
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileBasedDataSourceTest.scala
##########
@@ -52,7 +54,8 @@ private[sql] trait FileBasedDataSourceTest extends SQLTestUtils {
f(spark.read.format(dataSourceName).load(path.toString))
}
if (testVectorized) {
- withSQLConf(vectorizedReaderEnabledKey -> "true") {
+ withSQLConf(vectorizedReaderEnabledKey -> "true",
+ vectorizedReaderNestedEnabledKey -> "true") {
Review comment:
Sure. Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-990045189
@sadikovi do you have more comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r779180183
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
Yes, you've eliminated the schema lookup penalty.
There still seems to be the remaining 10-15% performance penalty. Earlier, I speculated on the cause of this particular penalty:
>this PR is still 10% slower than the baseline when testing the wide row case. I haven't confirmed it, but I think this is due to the extra OnHeapColumnVector instances created in ParquetColumnVector, for which you already have a TODO.
So maybe that will get resolved once the TODO is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r779225423
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
>how can I reproduce the 10-15% performance penalty
Maybe my old 2015 2.5 GHz MBP just feels it more strongly.
I launched spark-shell it with this command line
```
bin/spark-shell --driver-memory 5g --master "local[1]"
```
Maybe the 5g makes a difference (e.g., more pressure on the garbage collector when there are lots of OnHeapColumnVector instances).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973706494
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973830072
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760523416
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -42,24 +44,52 @@
/** The current row range */
private RowRange currentRange;
+ /** Maximum repetition level for the Parquet column */
+ final int maxRepetitionLevel;
+
/** Maximum definition level for the Parquet column */
final int maxDefinitionLevel;
+ /** Whether this column is required */
+ final boolean isRequired;
+
/** The current index over all rows within the column chunk. This is used to check if the
* current row should be skipped by comparing against the row ranges. */
long rowId;
- /** The offset in the current batch to put the next value */
- int offset;
+ /** The offset in the current batch to put the next value in value vector */
+ int valueOffset;
+
+ /** The offset in the current batch to put the next value in repetition & definition vector */
+ int levelOffset;
/** The remaining number of values to read in the current page */
int valuesToReadInPage;
- /** The remaining number of values to read in the current batch */
- int valuesToReadInBatch;
+ /** The remaining number of rows to read in the current batch */
+ int rowsToReadInBatch;
+
+
+ /* The following fields are only used when reading repeated values */
+
+ /** When processing repeated values, whether we've found the beginning of the first list after the
Review comment:
Update what?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984234595
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999762197
**[Test build #146490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146490/testReport)** for PR 34659 at commit [`77b2aec`](https://github.com/apache/spark/commit/77b2aec91350244dcf02489c2bc6f0d09f7bab9d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999823625
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50965/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r835016050
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel < maxElementDefinitionLevel) {
+ // This means the collection is null, and it is not an array element. In this case, we
+ // should increase offset to skip it when returning an array starting from the offset.
+ //
+ // For instance, considering an array of strings with 2 elements like the following:
+ // null, [a, b, c]
+ // the child array (which is of String type) in this case will be:
+ // null: 1 0 0 0
+ // length: 0 1 1 1
+ // offset: 0 0 1 2
+ // and the array itself will be:
+ // null: 1 0
+ // length: 0 3
+ // offset: 0 1
+ //
+ // It's important that for the second element `[a, b, c]`, the offset starts from 1
+ // since otherwise we'd include the first null element from child array in the result.
+ offset += 1;
+ }
+ if (definitionLevel <= maxDefinitionLevel - 1) {
+ // Collection is null or one of its optional parents is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // Collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // Collection is defined and non-empty: find out how many top elements are there until the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
Review comment:
nit: this seems not about offsets but adding nulls into struct and update row counts.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -707,15 +711,47 @@ public WritableColumnVector arrayData() {
public WritableColumnVector getChild(int ordinal) { return childColumns[ordinal]; }
/**
- * Returns the elements appended.
+ * Returns the number of child vectors.
+ */
+ public int getNumChildren() {
+ return childColumns.length;
+ }
+
+ /**
+ * Returns the elements appended. This is useful
*/
public final int getElementsAppended() { return elementsAppended; }
+ /**
+ * Increment number of elements appended by 'num'.
+ *
+ * This is useful when one wants to the 'putXXX' API to add new elements to the vector, but
Review comment:
```suggestion
* This is useful when one wants to use the 'putXXX' API to add new elements to the vector, but
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840769246
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -714,15 +718,47 @@ public WritableColumnVector arrayData() {
public WritableColumnVector getChild(int ordinal) { return childColumns[ordinal]; }
/**
- * Returns the elements appended.
+ * Returns the number of child vectors.
+ */
+ public int getNumChildren() {
+ return childColumns.length;
+ }
+
+ /**
+ * Returns the elements appended. This is useful
*/
public final int getElementsAppended() { return elementsAppended; }
+ /**
+ * Increment number of elements appended by 'num'.
+ *
+ * This is useful when one wants to use the 'putXXX' API to add new elements to the vector, but
+ * still want to keep count of how many elements have been added (since the 'putXXX' APIs don't
+ * increment count).
+ */
+ public final void addElementsAppended(int num) {
+ elementsAppended += num;
+ }
+
/**
* Marks this column as being constant.
*/
public final void setIsConstant() { isConstant = true; }
+ /**
+ * Marks this column only contains null values.
+ */
+ public final void setAllNull() {
Review comment:
This is similar to `setIsConstant` though: you can also call `column.putXXX` after the call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839092045
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
Review comment:
> Similarly for required element, maxElementDefinitionLevel is 2.
Yea, this is why I'm confused here. Seems it is when `child.required()`, element's def level is 2, but in the code, `maxElementDefinitionLevel` is 2 when `!child.required()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839100138
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
Review comment:
`readBatchRepeated` sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803329816
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
Review comment:
Should we lazy initializing `children` after line 86 and use `column.children()` to specify the `ArrayList` size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1035295235
Thanks @dongjoon-hyun @LuciferYang !
> Lastly, let's spin off (non-Parquet) ColumnVector layer as an independent PR before this PR.
Good suggestion. I think this part can be isolated. Let me do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1061870836
@timarmstrong can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r826414621
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // Values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // The current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // This RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // Need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // The current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // The range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // Skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // Read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
+ currentCount -= n;
+ leftInBatch -= n;
+ leftInPage -= n;
+ }
+ }
+ } else {
+ // Not a top-level row: just read all the repetition levels in the block if the row
+ // should be included according to row indexes, else skip the rows.
+ if (!state.shouldSkip) {
+ repLevels.appendInts(valuesLeftInBlock, currentValue);
+ }
+ state.numBatchedDefLevels += valuesLeftInBlock;
+ leftInPage -= valuesLeftInBlock;
+ currentCount -= valuesLeftInBlock;
+ }
+ break;
+ case PACKED:
+ int i = 0;
+
+ for (; i < valuesLeftInBlock; i++) {
+ int currentValue = currentBuffer[currentBufferIdx + i];
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ break;
+ } else if (rowId < rangeStart) {
+ // This is a top-level row, therefore check if we should skip it with row indexes
+ // the row is before the current range, skip it
+ defLevelProcessor.skipValues(1);
+ } else if (rowId > rangeEnd) {
+ // The row is after the current range, move to the next range and compare again
+ state.nextRange();
+ break;
+ } else {
+ // The row is in the current range, decrement the row counter and read it
+ leftInBatch--;
+ repLevels.appendInt(0);
+ defLevelProcessor.readValues(1);
+ }
+ rowId++;
+ } else {
+ if (!state.shouldSkip) {
+ repLevels.appendInt(currentValue);
+ }
+ state.numBatchedDefLevels += 1;
+ }
+ }
+
+ leftInPage -= i;
+ currentCount -= i;
+ currentBufferIdx += i;
+ break;
+ }
+ }
+
+ // Process all the batched def levels
+ defLevelProcessor.finish();
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ private static class DefLevelProcessor {
+ private final VectorizedRleValuesReader reader;
+ private final ParquetReadState state;
+ private final WritableColumnVector defLevels;
+ private final WritableColumnVector values;
+ private final WritableColumnVector nulls;
+ private final boolean valuesReused;
+ private final VectorizedValuesReader valueReader;
+ private final ParquetVectorUpdater updater;
+
+ DefLevelProcessor(
+ VectorizedRleValuesReader reader,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ this.reader = reader;
+ this.state = state;
+ this.defLevels = defLevels;
+ this.values = values;
+ this.nulls = nulls;
+ this.valuesReused = valuesReused;
+ this.valueReader = valueReader;
+ this.updater = updater;
+ }
+
+ void readValues(int n) {
+ if (!state.shouldSkip) {
+ state.numBatchedDefLevels += n;
Review comment:
This is to enable batch reading on the definition levels & values as much as possible. For instance, if we have the following triplets:
| repetition level | definition level | values |
----------------|---------------|--------
| 3 | 1 | 0 |
| 2 | 1 | 1 |
| 3 | 1 | 2 |
| 2 | 1 | 3 |
| 3 | 1 | 4 |
| 2 | 1 | 5 |
First we'd have to read the repetition levels, which must be encoded with bit-packing and therefore have to be read one by one. On the other hand, the definition levels are probably encoded in RLE and therefore, together with the values, can be read in one batch.
If we don't use this, then we'd have to call `reader.readValues` for each repetition level read here, which is more expensive I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984230804
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973781793
ACK. I will review it tomorrow or next week if that's okay 🙂.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r753865400
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java
##########
@@ -176,8 +176,7 @@ public void initBatch(
// Initialize the missing columns once.
if (colId == -1) {
OnHeapColumnVector missingCol = new OnHeapColumnVector(capacity, dt);
- missingCol.putNulls(0, capacity);
- missingCol.setIsConstant();
+ missingCol.setAllNull();
Review comment:
Oh, almost forgot to ask.
Is `missingCol.setAllNull();` equivalent to
```
missingCol.putNulls(0, capacity);
missingCol.setIsConstant();
```
? I thought `setAllNull` was an alias for the two method calls but it does not appear to be the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-982016495
Thanks @agrawaldevesh . Will address your comments soon.
> How do we confirm that this hasn't slowed down the non nested code path ?
I've run the `DataSourceReadBenchmark` through GitHub workflow with and without the PR, and put the results [here](https://gist.github.com/sunchao/fe67e6846b9fc2a46412b6de7e89afad). It seems the results don't change much but there are some variances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r774322647
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
>see what's the causing the perf issue in ParquetToSparkSchemaConverter#convertInternal
It's definitely the n\*n (or n\*(n/2) on average) loop here:
```
val converted = (0 until groupColumn.getChildrenCount).map { i =>
val field = groupColumn.getChild(i)
val fieldFromReadSchema = sparkReadSchema.flatMap { schema =>
schema.find(f => isSameFieldName(f.name, field.getName, caseSensitive))
}
```
`schema.find` should instead be a map lookup, or something like that.
>it'd be great if you already have some code snippet to share
Sure, in a verbose state:
```
// test for wide rows
val home_candidate = sys.env("HOME")
val home = if (!home_candidate.endsWith("/")) {
s"${home_candidate}/"
} else {
home_candidate
}
val width = 6000
val selectExpr = (1 to width).map { i =>
s"value as c$i"
}
import scala.util.Random
val r = new Random(65657652L)
val loopCount = 25
for (i <- 0 until loopCount) {
// println(s"iteration $i")
val df = spark.range(5000).map(_ => r.nextLong).toDF().selectExpr(selectExpr: _*)
val mode = if (i == 0) {
"overwrite"
} else {
"append"
}
df.coalesce(1).write.mode(mode).format("parquet").save(s"${home}data/wide_row_parquet")
}
// read test
val home_candidate = sys.env("HOME")
val home = if (!home_candidate.endsWith("/")) {
s"${home_candidate}/"
} else {
home_candidate
}
spark.read.parquet(s"${home}data/wide_row_parquet").createOrReplaceTempView("df")
val startTime = System.currentTimeMillis
sql("select * from df where (c5*c12) == 12").collect
(System.currentTimeMillis - startTime)/60.0/1000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r774322647
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
>see what's the causing the perf issue in ParquetToSparkSchemaConverter#convertInternal
It's definitely the n\*n (or n\*(n/2) on average) loop here:
```
val converted = (0 until groupColumn.getChildrenCount).map { i =>
val field = groupColumn.getChild(i)
val fieldFromReadSchema = sparkReadSchema.flatMap { schema =>
schema.find(f => isSameFieldName(f.name, field.getName, caseSensitive))
}
```
`schema.find` should instead be a map lookup, or something like that.
>it'd be great if you already have some code snippet to share
Sure, in a verbose state (edit: I ran with `--master "local[1]"`):
```
// test for wide rows
val home_candidate = sys.env("HOME")
val home = if (!home_candidate.endsWith("/")) {
s"${home_candidate}/"
} else {
home_candidate
}
val width = 6000
val selectExpr = (1 to width).map { i =>
s"value as c$i"
}
import scala.util.Random
val r = new Random(65657652L)
val loopCount = 25
for (i <- 0 until loopCount) {
// println(s"iteration $i")
val df = spark.range(5000).map(_ => r.nextLong).toDF().selectExpr(selectExpr: _*)
val mode = if (i == 0) {
"overwrite"
} else {
"append"
}
df.coalesce(1).write.mode(mode).format("parquet").save(s"${home}data/wide_row_parquet")
}
// read test
val home_candidate = sys.env("HOME")
val home = if (!home_candidate.endsWith("/")) {
s"${home_candidate}/"
} else {
home_candidate
}
spark.read.parquet(s"${home}data/wide_row_parquet").createOrReplaceTempView("df")
val startTime = System.currentTimeMillis
sql("select * from df where (c5*c12) == 12").collect
(System.currentTimeMillis - startTime)/60.0/1000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r834685472
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
Review comment:
```suggestion
import com.google.common.base.Preconditions;
import org.apache.spark.memory.MemoryMode;
```
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -738,6 +774,12 @@ public WritableColumnVector arrayData() {
*/
protected boolean isConstant;
+ /**
+ * True if this column only contains nulls. Comparing to 'isConstant' above, this doesn't require
+ * any allocation of space.
Review comment:
```suggestion
* True if this column only contains nulls. This means the column values never change, even
* across resets. Comparing to 'isConstant' above, this doesn't require
* any allocation of space.
```
Based on `reset` change.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
Review comment:
why don't update `state.levelOffset` in place like `valueOffset`? Could be more consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r834916330
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
Review comment:
Hmm, isn't that we should do `maxElementDefinitionLevel -= 1;` when `child.required()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r838980484
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -246,25 +249,25 @@ private void initBatch(
}
}
+ WritableColumnVector[] vectors;
if (memMode == MemoryMode.OFF_HEAP) {
- columnVectors = OffHeapColumnVector.allocateColumns(capacity, batchSchema);
+ vectors = OffHeapColumnVector.allocateColumns(capacity, batchSchema);
} else {
- columnVectors = OnHeapColumnVector.allocateColumns(capacity, batchSchema);
+ vectors = OnHeapColumnVector.allocateColumns(capacity, batchSchema);
+ }
+ columnarBatch = new ColumnarBatch(vectors);
+
+ columnVectors = new ParquetColumnVector[sparkSchema.fields().length];
+ for (int i = 0; i < columnVectors.length; i++) {
+ columnVectors[i] = new ParquetColumnVector(parquetColumn.children().apply(i),
Review comment:
The `parquetColumn` corresponds to the `StructType` for Spark's requested schema, so we need to apply on the children of it. Let me know if I understand your question correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840794435
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +215,379 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
}
}
- state.advanceOffsetAndRowId(offset, rowId);
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ /**
+ * Reads a batch of repetition levels, definition levels and values into 'repLevels',
+ * 'defLevels' and 'values' respectively. The definition levels and values are read via
+ * 'defLevelsReader' and 'valueReader' respectively.
+ * <p>
+ * The related states such as row index, offset, number of rows left in the batch and page,
+ * are tracked by 'state'. The type-specific 'updater' is used to update or skip values.
+ */
+ public void readBatchRepeated(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchRepeatedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ /**
+ * Reads a batch of repetition levels, definition levels and integer values into 'repLevels',
+ * 'defLevels', 'values' and 'nulls' respectively. The definition levels and values are read via
+ * 'defLevelsReader' and 'valueReader' respectively.
+ * <p>
+ * The 'values' vector is used to hold non-null values, while 'nulls' vector is used to hold
+ * null values.
+ * <p>
+ * The related states such as row index, offset, number of rows left in the batch and page,
+ * are tracked by 'state'.
+ * <p>
+ * Unlike 'readBatchRepeated', this is used to decode dictionary indices in dictionary encoding.
+ */
+ public void readIntegersRepeated(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchRepeatedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchRepeatedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // Values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // The current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // This RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // Need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // The current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // The range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // Skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // Read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
+ currentCount -= n;
+ leftInBatch -= n;
+ leftInPage -= n;
+ }
+ }
+ } else {
+ // Not a top-level row: just read all the repetition levels in the block if the row
+ // should be included according to row indexes, else skip the rows.
+ if (!state.shouldSkip) {
+ repLevels.appendInts(valuesLeftInBlock, currentValue);
+ }
+ state.numBatchedDefLevels += valuesLeftInBlock;
+ leftInPage -= valuesLeftInBlock;
+ currentCount -= valuesLeftInBlock;
+ }
+ break;
+ case PACKED:
+ int i = 0;
+
+ for (; i < valuesLeftInBlock; i++) {
+ int currentValue = currentBuffer[currentBufferIdx + i];
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ break;
+ } else if (rowId < rangeStart) {
+ // This is a top-level row, therefore check if we should skip it with row indexes
+ // the row is before the current range, skip it
+ defLevelProcessor.skipValues(1);
+ } else if (rowId > rangeEnd) {
+ // The row is after the current range, move to the next range and compare again
+ state.nextRange();
+ break;
+ } else {
+ // The row is in the current range, decrement the row counter and read it
+ leftInBatch--;
+ repLevels.appendInt(0);
+ defLevelProcessor.readValues(1);
+ }
+ rowId++;
+ } else {
+ if (!state.shouldSkip) {
+ repLevels.appendInt(currentValue);
+ }
+ state.numBatchedDefLevels += 1;
+ }
+ }
+
+ leftInPage -= i;
+ currentCount -= i;
+ currentBufferIdx += i;
+ break;
+ }
+ }
+
+ // Process all the batched def levels
+ defLevelProcessor.finish();
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ private static class DefLevelProcessor {
+ private final VectorizedRleValuesReader reader;
+ private final ParquetReadState state;
+ private final WritableColumnVector defLevels;
+ private final WritableColumnVector values;
+ private final WritableColumnVector nulls;
+ private final boolean valuesReused;
+ private final VectorizedValuesReader valueReader;
+ private final ParquetVectorUpdater updater;
+
+ DefLevelProcessor(
+ VectorizedRleValuesReader reader,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ this.reader = reader;
+ this.state = state;
+ this.defLevels = defLevels;
+ this.values = values;
+ this.nulls = nulls;
+ this.valuesReused = valuesReused;
+ this.valueReader = valueReader;
+ this.updater = updater;
+ }
+
+ void readValues(int n) {
+ if (!state.shouldSkip) {
+ state.numBatchedDefLevels += n;
+ } else {
+ reader.skipValues(state.numBatchedDefLevels, state, valueReader, updater);
+ state.numBatchedDefLevels = n;
+ state.shouldSkip = false;
+ }
+ }
+
+ void skipValues(int n) {
+ if (state.shouldSkip) {
+ state.numBatchedDefLevels += n;
+ } else {
+ reader.readValues(state.numBatchedDefLevels, state, defLevels, values, nulls, valuesReused,
+ valueReader, updater);
+ state.numBatchedDefLevels = n;
+ state.shouldSkip = true;
+ }
+ }
+
+ void finish() {
+ if (state.numBatchedDefLevels > 0) {
+ if (state.shouldSkip) {
+ reader.skipValues(state.numBatchedDefLevels, state, valueReader, updater);
+ } else {
+ reader.readValues(state.numBatchedDefLevels, state, defLevels, values, nulls,
+ valuesReused, valueReader, updater);
+ }
+ state.numBatchedDefLevels = 0;
+ }
+ }
+ }
+
+ /**
+ * Read the next 'total' values (either null or non-null) from this definition level reader and
+ * 'valueReader'. The definition levels are read into 'defLevels'. If a value is not
+ * null, it is appended to 'values'. Otherwise, a null bit will be set in 'nulls'.
+ *
+ * This is only used when reading repeated values.
+ */
+ private void readValues(
+ int total,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ defLevels.reserveAdditional(total);
+ values.reserveAdditional(total);
+ if (!valuesReused) {
+ // 'nulls' is a separate column vector so we'll have to reserve it separately
+ nulls.reserveAdditional(total);
+ }
+
+ int n = total;
+ int initialValueOffset = state.valueOffset;
+ while (n > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+ int num = Math.min(n, this.currentCount);
+ switch (mode) {
+ case RLE:
+ if (currentValue == state.maxDefinitionLevel) {
+ updater.readValues(num, state.valueOffset, values, valueReader);
+ state.valueOffset += num;
+ } else {
+ // Only add null if this represents a null element, but not the case when a
+ // collection is null or empty.
+ nulls.putNulls(state.valueOffset, num);
+ state.valueOffset += num;
+ }
+ defLevels.putInts(state.levelOffset, num, currentValue);
+ break;
+ case PACKED:
Review comment:
Let me split this into a util method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839123809
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -897,6 +897,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_VECTORIZED_READER_NESTED_COLUMN_ENABLED =
+ buildConf("spark.sql.parquet.enableNestedColumnVectorizedReader")
+ .doc("Enables vectorized Parquet decoding for nested columns (e.g., struct, list, map). " +
+ s"Note to enable this ${PARQUET_VECTORIZED_READER_ENABLED.key} also needs to be enabled.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
Make sure we remember to turn it off as default before merging. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r829408672
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
Review comment:
I have the follow-up to skip creating these column vectors and reading definition levels if unnecessary done [here](https://github.com/sunchao/spark/commit/f968791635db1d679638511fc4ff01d2b1eccff0). I can include it to the PR if it doesn't complicated the review :) cc @sadikovi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r838984112
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
Review comment:
Yes this is only for repeated values, e.g., array, map. I can rename it to `readBatchRepeated` if that's more clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1083908905
Thanks @viirya for reviewing! I've addressed your comments, and please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1086425510
FYI I plan to open a follow-up PR to disable the config and also update related tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya closed pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya closed pull request #34659:
URL: https://github.com/apache/spark/pull/34659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1020900026
@sunchao Is this pr still going on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r826423850
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -763,6 +805,10 @@ protected boolean isArray() {
DecimalType.isByteArrayDecimalType(type);
}
+ protected boolean isStruct() {
+ return type instanceof StructType || type instanceof CalendarIntervalType;
Review comment:
Hmm this is no longer used - I created it for the `WritableColumnVector` change I think, but now the change is no longer needed. Let me remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r769131591
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -350,6 +355,14 @@ public WritableColumnVector reserveDictionaryIds(int capacity) {
*/
public abstract void putArray(int rowId, int offset, int length);
+ /**
+ * Puts a new non-null struct at 'rowId' of this vector, which is backed by elements at
+ * 'offset' of child vectors.
+ *
+ * NOTE: this MUST be called after new elements are appended to child vectors of a struct vector.
+ */
+ public abstract void putStruct(int rowId, int offset);
Review comment:
I want to point out that this is a behavior change in `WritableColumnVector` so anyone who's using this to write struct will need to adopt to the new API.
Wanna see if you have any concern on this @cloud-fan @sadikovi .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r769139014
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -350,6 +355,14 @@ public WritableColumnVector reserveDictionaryIds(int capacity) {
*/
public abstract void putArray(int rowId, int offset, int length);
+ /**
+ * Puts a new non-null struct at 'rowId' of this vector, which is backed by elements at
+ * 'offset' of child vectors.
+ *
+ * NOTE: this MUST be called after new elements are appended to child vectors of a struct vector.
+ */
+ public abstract void putStruct(int rowId, int offset);
Review comment:
IMHO, it is fine as the PR goes to master, this is supposed to be internal API anyway unless I am mistaken 🙂.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984234595
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999820097
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50965/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r767332642
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
+ for (ParquetColumn column : JavaConverters.seqAsJavaList(parquetColumn.children())) {
+ checkColumn(column);
+ }
+ }
- String[] colPath = paths.get(i);
- if (fileSchema.containsPath(colPath)) {
- ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);
- if (!fd.equals(columns.get(i))) {
+ /**
+ * Check whether a column from requested schema is missing from the file schema, or whether it
+ * conforms to the type of the file schema.
+ */
+ private void checkColumn(ParquetColumn column) throws IOException {
+ String[] path = JavaConverters.seqAsJavaList(column.path()).toArray(new String[0]);
Review comment:
Yes, let's address separately.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
+ for (ParquetColumn column : JavaConverters.seqAsJavaList(parquetColumn.children())) {
+ checkColumn(column);
+ }
+ }
- String[] colPath = paths.get(i);
- if (fileSchema.containsPath(colPath)) {
- ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);
- if (!fd.equals(columns.get(i))) {
+ /**
+ * Check whether a column from requested schema is missing from the file schema, or whether it
+ * conforms to the type of the file schema.
+ */
+ private void checkColumn(ParquetColumn column) throws IOException {
+ String[] path = JavaConverters.seqAsJavaList(column.path()).toArray(new String[0]);
+ if (containsPath(fileSchema, path)) {
+ if (column.isPrimitive()) {
+ ColumnDescriptor desc = column.descriptor().get();
+ ColumnDescriptor fd = fileSchema.getColumnDescription(desc.getPath());
+ if (!fd.equals(desc)) {
throw new UnsupportedOperationException("Schema evolution not supported.");
}
- missingColumns[i] = false;
} else {
- if (columns.get(i).getMaxDefinitionLevel() == 0) {
- // Column is missing in data but the required data is non-nullable. This file is invalid.
- throw new IOException("Required column is missing in data file. Col: " +
- Arrays.toString(colPath));
+ for (ParquetColumn childColumn : JavaConverters.seqAsJavaList(column.children())) {
Review comment:
Same here, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973830072
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760528848
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
+ for (ParquetColumn column : JavaConverters.seqAsJavaList(parquetColumn.children())) {
+ checkColumn(column);
+ }
+ }
- String[] colPath = paths.get(i);
- if (fileSchema.containsPath(colPath)) {
- ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);
- if (!fd.equals(columns.get(i))) {
+ /**
+ * Check whether a column from requested schema is missing from the file schema, or whether it
+ * conforms to the type of the file schema.
+ */
+ private void checkColumn(ParquetColumn column) throws IOException {
+ String[] path = JavaConverters.seqAsJavaList(column.path()).toArray(new String[0]);
Review comment:
It could .. although I'd have to change a lot of places in `ParquetSchemaSuite`. Since this is a relatively small issue, maybe let me address it separately?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760568934
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java
##########
@@ -176,8 +176,7 @@ public void initBatch(
// Initialize the missing columns once.
if (colId == -1) {
OnHeapColumnVector missingCol = new OnHeapColumnVector(capacity, dt);
- missingCol.putNulls(0, capacity);
- missingCol.setIsConstant();
+ missingCol.setAllNull();
Review comment:
Not quite the same. `setIsConstant` requires to allocate space for the constant elements, but `setAllNull` doesn't. This is useful for those missing columns that are of nested types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] klaus-xiong commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
klaus-xiong commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r783793553
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -103,9 +104,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
fileReader.setRequestedSchema(requestedSchema);
String sparkRequestedSchemaString =
configuration.get(ParquetReadSupport$.MODULE$.SPARK_ROW_REQUESTED_SCHEMA());
- this.sparkSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ StructType sparkRequestedSchema = StructType$.MODULE$.fromString(sparkRequestedSchemaString);
+ ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(configuration);
+ this.parquetColumn = converter.convertParquetColumn(requestedSchema,
+ Option.apply(sparkRequestedSchema));
+ this.sparkSchema = (StructType) parquetColumn.sparkType();
this.totalRowCount = fileReader.getFilteredRecordCount();
+
Review comment:
it is a redundant blank line here?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
##########
@@ -169,8 +170,8 @@ class ParquetFileFormat
override def supportBatch(sparkSession: SparkSession, schema: StructType): Boolean = {
val conf = sparkSession.sessionState.conf
conf.parquetVectorizedReaderEnabled && conf.wholeStageEnabled &&
- schema.length <= conf.wholeStageMaxNumFields &&
- schema.forall(_.dataType.isInstanceOf[AtomicType])
+ ParquetUtils.isBatchReadSupportedForSchema(conf, schema) &&
+ !WholeStageCodegenExec.isTooManyFields(conf, schema)
Review comment:
indent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803980475
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -144,6 +145,30 @@ object ParquetUtils {
file.getName == ParquetFileWriter.PARQUET_METADATA_FILE
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
Review comment:
Makes sense. Let me change it.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -144,6 +145,30 @@ object ParquetUtils {
file.getName == ParquetFileWriter.PARQUET_METADATA_FILE
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ schema.forall(f => isBatchReadSupported(sqlConf, f.dataType))
+
+ def isBatchReadSupported(sqlConf: SQLConf, dt: DataType): Boolean = dt match {
Review comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1034226787
Could you review this? @cloud-fan @dongjoon-hyun @viirya ? It'd be great if this can catch Spark 3.3 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803330689
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -144,6 +145,30 @@ object ParquetUtils {
file.getName == ParquetFileWriter.PARQUET_METADATA_FILE
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
Review comment:
It would be great to design new functions minimally which receive the minimal information instead of passing over the full `SQLConf` recursively. For example, could you revise
- `isBatchReadSupportedForSchema` by handing over the values of `sqlConf.parquetVectorizedReaderEnabled` and `sqlConf.parquetVectorizedReaderNestedColumnEnabled`
- `isBatchReadSupported` by handing over the value of `sqlConf.parquetVectorizedReaderNestedColumnEnabled` from `isBatchReadSupportedForSchema`.
directly instead of `SQLConf` recursively?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r767332599
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
Review comment:
All good, I remember Java 6 used to complain if you use `<>` diamond type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998400058
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50886/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-979373931
Thanks @sadikovi , really appreciate your feedback! will address the comments soon! and then you can re-visit the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-978875609
@sunchao Let me know if you would like me to revisit the code in this PR. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973675452
**[Test build #145429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145429/testReport)** for PR 34659 at commit [`10c3a45`](https://github.com/apache/spark/commit/10c3a450af62b0028601c862005a566d623af253).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973722372
cc @sadikovi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973736109
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760524332
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -39,12 +42,13 @@
import org.apache.spark.sql.vectorized.ColumnarBatch;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
+import scala.collection.JavaConverters;
Review comment:
Yes it's used below in `checkColumn` method for example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760653817
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
+ if (v == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
+ } else if (!state.isRequired && v == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, v);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // the current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // this RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // the current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // the range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ leftInBatch -= n;
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
Review comment:
Yes better to move it here. The member vars (e.g., `currentCount`) tracks the state of the RLE/BitPacked block being read, while the local vars track the state of the input vectors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r838972041
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -157,7 +160,11 @@ private boolean isLazyDecodingSupported(PrimitiveType.PrimitiveTypeName typeName
/**
* Reads `total` values from this columnReader into column.
Review comment:
Yes this is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839073014
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
Review comment:
Oh, I see. `maxDefinitionLevel` is confused me that as I thought it is max def level for elements. Because this equation `definitionLevel > maxDefinitionLevel` looks not reasonable. Actually you indicate two def levels at different element (array and nested element).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839059936
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // Values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // The current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // This RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // Need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // The current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // The range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // Skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // Read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
+ currentCount -= n;
+ leftInBatch -= n;
+ leftInPage -= n;
+ }
+ }
+ } else {
+ // Not a top-level row: just read all the repetition levels in the block if the row
+ // should be included according to row indexes, else skip the rows.
+ if (!state.shouldSkip) {
+ repLevels.appendInts(valuesLeftInBlock, currentValue);
+ }
+ state.numBatchedDefLevels += valuesLeftInBlock;
Review comment:
Yes, the definition levels and values are read in `DefLevelProcessor`, in a batched fashion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839059318
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
Review comment:
No, it reads definition levels & values too. It's just the number of rep_level/def_level/value to read is controlled by repetition levels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840639102
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -187,6 +188,30 @@ object ParquetUtils {
}
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ schema.forall(f => isBatchReadSupported(sqlConf, f.dataType))
+
+ def isBatchReadSupported(sqlConf: SQLConf, dt: DataType): Boolean = dt match {
+ case _: AtomicType =>
+ true
+ case at: ArrayType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, at.elementType)
+ case mt: MapType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, mt.keyType) &&
+ isBatchReadSupported(sqlConf, mt.valueType)
+ case st: StructType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ st.fields.forall(f => isBatchReadSupported(sqlConf, f.dataType))
Review comment:
nit: should be 2 spaces here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1020900026
@sunchao Is this pr still going on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999762197
**[Test build #146490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146490/testReport)** for PR 34659 at commit [`77b2aec`](https://github.com/apache/spark/commit/77b2aec91350244dcf02489c2bc6f0d09f7bab9d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803329816
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
Review comment:
Should we lazy initializing `children` after line 86 and use `column.children()` to specify the `ArrayList` initialCapacity, and in `isPrimitive` branch `children` can be assigned to `Collections.emptyList()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803334124
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java
##########
@@ -176,8 +176,7 @@ public void initBatch(
// Initialize the missing columns once.
if (colId == -1) {
OnHeapColumnVector missingCol = new OnHeapColumnVector(capacity, dt);
- missingCol.putNulls(0, capacity);
- missingCol.setIsConstant();
+ missingCol.setAllNull();
Review comment:
Is this related this PR or new test coverage?
This change looks like relying on the existing ORC tests only.
In that case, shall we make a separate PR for this ORC change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1035358198
@dongjoon-hyun I opened #35483 for the separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r753860971
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("can't set reader for non-primitive column");
Review comment:
nit: `Can't set...`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
Review comment:
I understand it requires a lot of typing but it would be good if we could add a small javadoc here and below to highlight that the methods do.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parent is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel == maxDefinitionLevel - 1) {
+ // the collection is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // collection is defined and non-empty: find out how many top element there is till the
Review comment:
nit: find out how many elements are there until the start of the next array. Alternatively, find out the number of elements until the start of the next array.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -897,6 +897,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_VECTORIZED_READER_NESTED_COLUMN_ENABLED =
+ buildConf("spark.sql.parquet.enableNestedColumnVectorizedReader")
+ .doc("Enables vectorized Parquet decoding for nested columns (e.g., struct, list, map). " +
+ s"Note to enable this ${PARQUET_VECTORIZED_READER_ENABLED.key} also needs to be enabled.")
Review comment:
I am not sure what the guidelines are on a conf text but maybe this would sound a bit crisper:
`Enables vectorized Parquet decoding for nested columns (e.g., struct, list, map). Requires ${PARQUET_VECTORIZED_READER_ENABLED.key} to be enabled.`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parent is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel == maxDefinitionLevel - 1) {
+ // the collection is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // collection is defined and non-empty: find out how many top element there is till the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i + 1);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
+ int maxRepetitionLevel = column.repetitionLevel();
+ int maxDefinitionLevel = column.definitionLevel();
+
+ vector.reserve(definitionLevels.getElementsAppended());
+
+ int rowId = 0;
+ int nonnullRowId = 0;
+ boolean hasRepetitionLevels = repetitionLevels.getElementsAppended() > 0;
+ for (int i = 0; i < definitionLevels.getElementsAppended(); i++) {
+ // if repetition level > maxRepetitionLevel, the value is a nested element (e.g., an array
+ // element in struct<array<int>>), and we should skip the definition level since it doesn't
+ // represent with the struct.
+ if (!hasRepetitionLevels || repetitionLevels.getInt(i) <= maxRepetitionLevel) {
+ if (definitionLevels.getInt(i) == maxDefinitionLevel - 1) {
+ // the struct is null
+ vector.putNull(rowId);
+ rowId++;
+ } else if (definitionLevels.getInt(i) >= maxDefinitionLevel) {
+ vector.putNotNull(rowId);
+ vector.putStruct(rowId, nonnullRowId);
+ rowId++;
+ nonnullRowId++;
+ }
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private static WritableColumnVector allocateLevelsVector(int capacity, MemoryMode memoryMode) {
+ switch (memoryMode) {
+ case ON_HEAP:
+ return new OnHeapColumnVector(capacity, DataTypes.IntegerType);
+ case OFF_HEAP:
+ return new OffHeapColumnVector(capacity, DataTypes.IntegerType);
+ default:
+ throw new IllegalArgumentException("Unknown memory mode: " + memoryMode);
+ }
+ }
+
+ private int getNextCollectionStart(int maxRepetitionLevel, int elementIndex) {
+ int idx = elementIndex + 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
+ break;
+ }
+ }
+ return idx;
+ }
+
+ private int getCollectionSize(int maxRepetitionLevel, int idx) {
Review comment:
We might need javadoc for this and a couple of methods above to go along with the inline comments.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -42,24 +44,52 @@
/** The current row range */
private RowRange currentRange;
+ /** Maximum repetition level for the Parquet column */
+ final int maxRepetitionLevel;
+
/** Maximum definition level for the Parquet column */
final int maxDefinitionLevel;
+ /** Whether this column is required */
+ final boolean isRequired;
+
/** The current index over all rows within the column chunk. This is used to check if the
* current row should be skipped by comparing against the row ranges. */
long rowId;
- /** The offset in the current batch to put the next value */
- int offset;
+ /** The offset in the current batch to put the next value in value vector */
+ int valueOffset;
+
+ /** The offset in the current batch to put the next value in repetition & definition vector */
+ int levelOffset;
/** The remaining number of values to read in the current page */
int valuesToReadInPage;
- /** The remaining number of values to read in the current batch */
- int valuesToReadInBatch;
+ /** The remaining number of rows to read in the current batch */
+ int rowsToReadInBatch;
+
+
+ /* The following fields are only used when reading repeated values */
+
+ /** When processing repeated values, whether we've found the beginning of the first list after the
+ * current batch. */
+ boolean lastListCompleted;
- ParquetReadState(int maxDefinitionLevel, PrimitiveIterator.OfLong rowIndexes) {
- this.maxDefinitionLevel = maxDefinitionLevel;
+ /** When processing repeated types, the number of accumulated definition levels to process */
+ int numBatchedDefLevels;
+
+ /** When processing repeated types, whether we should skip the current batch of definition
Review comment:
Same here.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parent is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel == maxDefinitionLevel - 1) {
+ // the collection is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // collection is defined and non-empty: find out how many top element there is till the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i + 1);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
+ int maxRepetitionLevel = column.repetitionLevel();
+ int maxDefinitionLevel = column.definitionLevel();
+
+ vector.reserve(definitionLevels.getElementsAppended());
+
+ int rowId = 0;
+ int nonnullRowId = 0;
+ boolean hasRepetitionLevels = repetitionLevels.getElementsAppended() > 0;
+ for (int i = 0; i < definitionLevels.getElementsAppended(); i++) {
+ // if repetition level > maxRepetitionLevel, the value is a nested element (e.g., an array
+ // element in struct<array<int>>), and we should skip the definition level since it doesn't
+ // represent with the struct.
+ if (!hasRepetitionLevels || repetitionLevels.getInt(i) <= maxRepetitionLevel) {
+ if (definitionLevels.getInt(i) == maxDefinitionLevel - 1) {
+ // the struct is null
+ vector.putNull(rowId);
+ rowId++;
+ } else if (definitionLevels.getInt(i) >= maxDefinitionLevel) {
+ vector.putNotNull(rowId);
+ vector.putStruct(rowId, nonnullRowId);
+ rowId++;
+ nonnullRowId++;
+ }
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private static WritableColumnVector allocateLevelsVector(int capacity, MemoryMode memoryMode) {
+ switch (memoryMode) {
+ case ON_HEAP:
+ return new OnHeapColumnVector(capacity, DataTypes.IntegerType);
+ case OFF_HEAP:
+ return new OffHeapColumnVector(capacity, DataTypes.IntegerType);
+ default:
+ throw new IllegalArgumentException("Unknown memory mode: " + memoryMode);
+ }
+ }
+
+ private int getNextCollectionStart(int maxRepetitionLevel, int elementIndex) {
+ int idx = elementIndex + 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
+ break;
+ }
+ }
+ return idx;
+ }
+
+ private int getCollectionSize(int maxRepetitionLevel, int idx) {
+ int size = 1;
Review comment:
I think `size = 1` requires a comment. Is it due to the fact that the collection is non-empty and we automatically add the first element?
Depending on the idx, we could double count size or skip elements. I would propose to return whatever size was calculated based on index and add +1 or +K in the caller's code if required.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
Review comment:
I don't know if Spark runs java lint. Is it allowed to use `<>` with generic types?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -303,55 +313,88 @@ public boolean nextBatch() throws IOException {
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
- // Check that the requested schema is supported.
- missingColumns = new boolean[requestedSchema.getFieldCount()];
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<String[]> paths = requestedSchema.getPaths();
- for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
- Type t = requestedSchema.getFields().get(i);
- if (!t.isPrimitive() || t.isRepetition(Type.Repetition.REPEATED)) {
- throw new UnsupportedOperationException("Complex types not supported.");
- }
+ missingColumns = new HashSet<>();
+ for (ParquetColumn column : JavaConverters.seqAsJavaList(parquetColumn.children())) {
+ checkColumn(column);
+ }
+ }
- String[] colPath = paths.get(i);
- if (fileSchema.containsPath(colPath)) {
- ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);
- if (!fd.equals(columns.get(i))) {
+ /**
+ * Check whether a column from requested schema is missing from the file schema, or whether it
+ * conforms to the type of the file schema.
+ */
+ private void checkColumn(ParquetColumn column) throws IOException {
+ String[] path = JavaConverters.seqAsJavaList(column.path()).toArray(new String[0]);
Review comment:
Maybe the original path could be returned as an array...
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -39,12 +42,13 @@
import org.apache.spark.sql.vectorized.ColumnarBatch;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
+import scala.collection.JavaConverters;
Review comment:
Is this import required?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -42,24 +44,52 @@
/** The current row range */
private RowRange currentRange;
+ /** Maximum repetition level for the Parquet column */
+ final int maxRepetitionLevel;
+
/** Maximum definition level for the Parquet column */
final int maxDefinitionLevel;
+ /** Whether this column is required */
+ final boolean isRequired;
+
/** The current index over all rows within the column chunk. This is used to check if the
* current row should be skipped by comparing against the row ranges. */
long rowId;
- /** The offset in the current batch to put the next value */
- int offset;
+ /** The offset in the current batch to put the next value in value vector */
+ int valueOffset;
+
+ /** The offset in the current batch to put the next value in repetition & definition vector */
+ int levelOffset;
/** The remaining number of values to read in the current page */
int valuesToReadInPage;
- /** The remaining number of values to read in the current batch */
- int valuesToReadInBatch;
+ /** The remaining number of rows to read in the current batch */
+ int rowsToReadInBatch;
+
+
+ /* The following fields are only used when reading repeated values */
+
+ /** When processing repeated values, whether we've found the beginning of the first list after the
Review comment:
Javadoc should be updated.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parent is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel == maxDefinitionLevel - 1) {
+ // the collection is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // collection is defined and non-empty: find out how many top element there is till the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i + 1);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
+ int maxRepetitionLevel = column.repetitionLevel();
+ int maxDefinitionLevel = column.definitionLevel();
+
+ vector.reserve(definitionLevels.getElementsAppended());
+
+ int rowId = 0;
+ int nonnullRowId = 0;
+ boolean hasRepetitionLevels = repetitionLevels.getElementsAppended() > 0;
+ for (int i = 0; i < definitionLevels.getElementsAppended(); i++) {
+ // if repetition level > maxRepetitionLevel, the value is a nested element (e.g., an array
+ // element in struct<array<int>>), and we should skip the definition level since it doesn't
+ // represent with the struct.
+ if (!hasRepetitionLevels || repetitionLevels.getInt(i) <= maxRepetitionLevel) {
+ if (definitionLevels.getInt(i) == maxDefinitionLevel - 1) {
+ // the struct is null
+ vector.putNull(rowId);
+ rowId++;
+ } else if (definitionLevels.getInt(i) >= maxDefinitionLevel) {
+ vector.putNotNull(rowId);
+ vector.putStruct(rowId, nonnullRowId);
+ rowId++;
+ nonnullRowId++;
+ }
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private static WritableColumnVector allocateLevelsVector(int capacity, MemoryMode memoryMode) {
+ switch (memoryMode) {
+ case ON_HEAP:
+ return new OnHeapColumnVector(capacity, DataTypes.IntegerType);
+ case OFF_HEAP:
+ return new OffHeapColumnVector(capacity, DataTypes.IntegerType);
+ default:
+ throw new IllegalArgumentException("Unknown memory mode: " + memoryMode);
+ }
+ }
+
+ private int getNextCollectionStart(int maxRepetitionLevel, int elementIndex) {
+ int idx = elementIndex + 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
+ break;
+ }
+ }
+ return idx;
+ }
+
+ private int getCollectionSize(int maxRepetitionLevel, int idx) {
+ int size = 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
+ break;
+ } else if (repetitionLevels.getInt(idx) <= maxRepetitionLevel + 1) {
+ // only count elements which belong to the current collection
+ // For instance, suppose we have the following Parquet schema:
+ //
+ // message schema { max rl max dl
+ // optional group col (LIST) { 0 1
+ // repeated group list { 1 2
+ // optional group element (LIST) { 1 3
+ // repeated group list { 2 4
+ // required int32 element; 2 4
+ // }
+ // }
+ // }
+ // }
+ // }
+ //
+ // For a list such as: [[[0, 1], [2, 3]], [[4, 5], [6, 7]]], the repetition & definition
+ // levels would be:
+ //
+ // repetition levels: [0, 2, 1, 2, 0, 2, 1, 2]
+ // definition levels: [2, 2, 2, 2, 2, 2, 2, 2]
+ //
+ // when calculating collection size for the outer array, we should only count repetition
+ // levels whose value is <= 1 (which is the max repetition level for the inner array)
Review comment:
Could you elaborate on this comment a bit more? Is it about the example above or in general?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-973675452
**[Test build #145429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145429/testReport)** for PR 34659 at commit [`10c3a45`](https://github.com/apache/spark/commit/10c3a450af62b0028601c862005a566d623af253).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r772709075
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -350,6 +355,14 @@ public WritableColumnVector reserveDictionaryIds(int capacity) {
*/
public abstract void putArray(int rowId, int offset, int length);
+ /**
+ * Puts a new non-null struct at 'rowId' of this vector, which is backed by elements at
+ * 'offset' of child vectors.
+ *
+ * NOTE: this MUST be called after new elements are appended to child vectors of a struct vector.
+ */
+ public abstract void putStruct(int rowId, int offset);
Review comment:
cool, thanks for confirming!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r771851647
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
##########
@@ -169,8 +170,8 @@ class ParquetFileFormat
override def supportBatch(sparkSession: SparkSession, schema: StructType): Boolean = {
val conf = sparkSession.sessionState.conf
conf.parquetVectorizedReaderEnabled && conf.wholeStageEnabled &&
- schema.length <= conf.wholeStageMaxNumFields &&
- schema.forall(_.dataType.isInstanceOf[AtomicType])
+ ParquetFileFormat.isBatchReadSupported(conf, schema) &&
Review comment:
When ```spark.sql.parquet.enableNestedColumnVectorizedReader=false```, which will be the normal case initially, I think returningBatch will always be false, even for non-complex cases, since this [line](https://github.com/apache/spark/blob/7b6e083833c807e44b76bc321d32e9c5857e648f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L249) always passes a struct (resultSchema) to ```supportBatch```.
At least, that's what my testing seems to show. With this PR and ```spark.sql.parquet.enableNestedColumnVectorizedReader=false```, returningBatch is false for a non-complex case. But on master, returningBatch is true for the same test:
```
spark.range(0, 10).map { x => (x, x + 1, x + 3) }.toDF("a", "b", "c").
write.mode("overwrite").format("parquet").save("simple_parquet")
sql("select a, b from `parquet`.`simple_parquet`").collect
```
and...
```
bash-3.2$ git diff | cat
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
index bd8d11d827..96182aafa9 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
@@ -340,6 +340,7 @@ class ParquetFileFormat
vectorizedReader.initialize(split, hadoopAttemptContext)
logDebug(s"Appending $partitionSchema ${file.partitionValues}")
vectorizedReader.initBatch(partitionSchema, file.partitionValues)
+ print(s"returningBatch is $returningBatch\n")
if (returningBatch) {
vectorizedReader.enableReturningBatches()
}
bash-3.2$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760484066
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
Review comment:
Sure, added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760630677
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
+ if (v == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
+ } else if (!state.isRequired && v == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, v);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // the current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // this RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // the current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // the range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ leftInBatch -= n;
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
Review comment:
Hmm what do you mean? we need to read in repetition levels in this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760632213
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
+ if (v == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
+ } else if (!state.isRequired && v == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, v);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
Review comment:
Yea good point, let me change the non nested code, although I think for that case `readNextGroup` will never return false since we know exactly how many elements to read beforehand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984300866
**[Test build #145831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145831/testReport)** for PR 34659 at commit [`7b6e083`](https://github.com/apache/spark/commit/7b6e083833c807e44b76bc321d32e9c5857e648f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760654420
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -897,6 +897,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_VECTORIZED_READER_NESTED_COLUMN_ENABLED =
+ buildConf("spark.sql.parquet.enableNestedColumnVectorizedReader")
+ .doc("Enables vectorized Parquet decoding for nested columns (e.g., struct, list, map). " +
+ s"Note to enable this ${PARQUET_VECTORIZED_READER_ENABLED.key} also needs to be enabled.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
This is to make sure we can pass all Parquet related tests :) I plan to turn it off later separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999936960
**[Test build #146490 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146490/testReport)** for PR 34659 at commit [`77b2aec`](https://github.com/apache/spark/commit/77b2aec91350244dcf02489c2bc6f0d09f7bab9d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803968556
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileBasedDataSourceTest.scala
##########
@@ -52,7 +54,8 @@ private[sql] trait FileBasedDataSourceTest extends SQLTestUtils {
f(spark.read.format(dataSourceName).load(path.toString))
}
if (testVectorized) {
- withSQLConf(vectorizedReaderEnabledKey -> "true") {
+ withSQLConf(vectorizedReaderEnabledKey -> "true",
+ vectorizedReaderNestedEnabledKey -> "true") {
Review comment:
Sure, good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sadikovi commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r824584947
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // Values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // The current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // This RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // Need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // The current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // The range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // Skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // Read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
+ currentCount -= n;
+ leftInBatch -= n;
+ leftInPage -= n;
+ }
+ }
+ } else {
+ // Not a top-level row: just read all the repetition levels in the block if the row
+ // should be included according to row indexes, else skip the rows.
+ if (!state.shouldSkip) {
+ repLevels.appendInts(valuesLeftInBlock, currentValue);
+ }
+ state.numBatchedDefLevels += valuesLeftInBlock;
+ leftInPage -= valuesLeftInBlock;
+ currentCount -= valuesLeftInBlock;
+ }
+ break;
+ case PACKED:
+ int i = 0;
+
+ for (; i < valuesLeftInBlock; i++) {
+ int currentValue = currentBuffer[currentBufferIdx + i];
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ break;
+ } else if (rowId < rangeStart) {
+ // This is a top-level row, therefore check if we should skip it with row indexes
+ // the row is before the current range, skip it
+ defLevelProcessor.skipValues(1);
+ } else if (rowId > rangeEnd) {
+ // The row is after the current range, move to the next range and compare again
+ state.nextRange();
+ break;
+ } else {
+ // The row is in the current range, decrement the row counter and read it
+ leftInBatch--;
+ repLevels.appendInt(0);
+ defLevelProcessor.readValues(1);
+ }
+ rowId++;
+ } else {
+ if (!state.shouldSkip) {
+ repLevels.appendInt(currentValue);
+ }
+ state.numBatchedDefLevels += 1;
+ }
+ }
+
+ leftInPage -= i;
+ currentCount -= i;
+ currentBufferIdx += i;
+ break;
+ }
+ }
+
+ // Process all the batched def levels
+ defLevelProcessor.finish();
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ private static class DefLevelProcessor {
+ private final VectorizedRleValuesReader reader;
+ private final ParquetReadState state;
+ private final WritableColumnVector defLevels;
+ private final WritableColumnVector values;
+ private final WritableColumnVector nulls;
+ private final boolean valuesReused;
+ private final VectorizedValuesReader valueReader;
+ private final ParquetVectorUpdater updater;
+
+ DefLevelProcessor(
+ VectorizedRleValuesReader reader,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ this.reader = reader;
+ this.state = state;
+ this.defLevels = defLevels;
+ this.values = values;
+ this.nulls = nulls;
+ this.valuesReused = valuesReused;
+ this.valueReader = valueReader;
+ this.updater = updater;
+ }
+
+ void readValues(int n) {
+ if (!state.shouldSkip) {
+ state.numBatchedDefLevels += n;
Review comment:
Apologies for a potentially dumb question, but why don't we call `reader.readValues(...)` here?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -39,12 +42,13 @@
import org.apache.spark.sql.vectorized.ColumnarBatch;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
+import scala.collection.JavaConverters;
Review comment:
That should probably be moved after java imports 🙂. I am probably nit-picking here but maybe it would be a good nit to fix - it is up to you.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
Review comment:
I would like to confirm that this check is by reference, e.g. if there are two instances of ParquetColumn with the same name and only one is in the set, the other one would return `false` when checking. If that is expected, it is all good.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -763,6 +805,10 @@ protected boolean isArray() {
DecimalType.isByteArrayDecimalType(type);
}
+ protected boolean isStruct() {
+ return type instanceof StructType || type instanceof CalendarIntervalType;
Review comment:
Do you know if we have tests for this? I am curious what happens if someone adds a union type or another composite type, will we need to update this method?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -187,6 +188,30 @@ object ParquetUtils {
}
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ schema.forall(f => isBatchReadSupported(sqlConf, f.dataType))
+
+ def isBatchReadSupported(sqlConf: SQLConf, dt: DataType): Boolean = dt match {
+ case _: AtomicType =>
+ true
+ case at: ArrayType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, at.elementType)
+ case mt: MapType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, mt.keyType) &&
+ isBatchReadSupported(sqlConf, mt.valueType)
+ case st: StructType =>
Review comment:
Should we support CalendarIntervalType here as well? I saw a method checking on that type also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r834907682
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel < maxElementDefinitionLevel) {
+ // This means the collection is null, and it is not an array element. In this case, we
+ // should increase offset to skip it when returning an array starting from the offset.
+ //
+ // For instance, considering an array of strings with 2 elements like the following:
+ // null, [a, b, c]
+ // the child array (which is of String type) in this case will be:
+ // null: 1 0 0 0
+ // length: 0 1 1 1
+ // offset: 0 0 1 2
+ // and the array itself will be:
+ // null: 1 0
+ // length: 0 3
+ // offset: 0 1
+ //
+ // It's important that for the second element `[a, b, c]`, the offset starts from 1
+ // since otherwise we'd include the first null element from child array in the result.
+ offset += 1;
+ }
+ if (definitionLevel <= maxDefinitionLevel - 1) {
+ // Collection is null or one of its optional parents is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // Collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // Collection is defined and non-empty: find out how many top elements are there until the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
+ int maxRepetitionLevel = column.repetitionLevel();
+ int maxDefinitionLevel = column.definitionLevel();
+
+ vector.reserve(definitionLevels.getElementsAppended());
+
+ int rowId = 0;
+ boolean hasRepetitionLevels = repetitionLevels.getElementsAppended() > 0;
+ for (int i = 0; i < definitionLevels.getElementsAppended(); i++) {
+ // If repetition level > maxRepetitionLevel, the value is a nested element (e.g., an array
+ // element in struct<array<int>>), and we should skip the definition level since it doesn't
+ // represent with the struct.
+ if (!hasRepetitionLevels || repetitionLevels.getInt(i) <= maxRepetitionLevel) {
+ if (definitionLevels.getInt(i) <= maxDefinitionLevel - 1) {
+ // Struct is null
+ vector.putNull(rowId);
+ rowId++;
+ } else if (definitionLevels.getInt(i) >= maxDefinitionLevel) {
+ vector.putNotNull(rowId);
+ rowId++;
+ }
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private static WritableColumnVector allocateLevelsVector(int capacity, MemoryMode memoryMode) {
+ switch (memoryMode) {
+ case ON_HEAP:
+ return new OnHeapColumnVector(capacity, DataTypes.IntegerType);
+ case OFF_HEAP:
+ return new OffHeapColumnVector(capacity, DataTypes.IntegerType);
+ default:
+ throw new IllegalArgumentException("Unknown memory mode: " + memoryMode);
+ }
+ }
+
+ /**
+ * For a collection (i.e., array or map) element at index 'idx', returns the starting index of
+ * the next collection after it.
+ *
+ * @param maxRepetitionLevel the maximum repetition level for the elements in this collection
+ * @param idx the index of this collection in the Parquet column
+ * @return the starting index of the next collection
+ */
+ private int getNextCollectionStart(int maxRepetitionLevel, int idx) {
+ idx += 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
Review comment:
If it is `maxRepetitionLevel`, `idx` is still repeated at the same array, no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r838971025
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
Review comment:
This is because `valueOffset` is only updated when the value is not null, but `levelOffset` is updated in all the cases. I could move this to the end of each `case` clause above, but I feel it's not too helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839077789
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel < maxElementDefinitionLevel) {
+ // This means the collection is null, and it is not an array element. In this case, we
+ // should increase offset to skip it when returning an array starting from the offset.
+ //
+ // For instance, considering an array of strings with 2 elements like the following:
+ // null, [a, b, c]
+ // the child array (which is of String type) in this case will be:
+ // null: 1 0 0 0
+ // length: 0 1 1 1
+ // offset: 0 0 1 2
+ // and the array itself will be:
+ // null: 1 0
+ // length: 0 3
+ // offset: 0 1
+ //
+ // It's important that for the second element `[a, b, c]`, the offset starts from 1
+ // since otherwise we'd include the first null element from child array in the result.
+ offset += 1;
+ }
+ if (definitionLevel <= maxDefinitionLevel - 1) {
Review comment:
Yes we can merge them (just need to add `offset += 1` in the first two if branches. I separated it out mainly to add the block of comments explaining why we need to increment `offset`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1083924110
Thanks @sunchao. I will take another look tonight or tomorrow early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839100438
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -246,25 +249,25 @@ private void initBatch(
}
}
+ WritableColumnVector[] vectors;
if (memMode == MemoryMode.OFF_HEAP) {
- columnVectors = OffHeapColumnVector.allocateColumns(capacity, batchSchema);
+ vectors = OffHeapColumnVector.allocateColumns(capacity, batchSchema);
} else {
- columnVectors = OnHeapColumnVector.allocateColumns(capacity, batchSchema);
+ vectors = OnHeapColumnVector.allocateColumns(capacity, batchSchema);
+ }
+ columnarBatch = new ColumnarBatch(vectors);
+
+ columnVectors = new ParquetColumnVector[sparkSchema.fields().length];
+ for (int i = 0; i < columnVectors.length; i++) {
+ columnVectors[i] = new ParquetColumnVector(parquetColumn.children().apply(i),
Review comment:
Oh, I see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839080186
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel < maxElementDefinitionLevel) {
+ // This means the collection is null, and it is not an array element. In this case, we
+ // should increase offset to skip it when returning an array starting from the offset.
+ //
+ // For instance, considering an array of strings with 2 elements like the following:
+ // null, [a, b, c]
+ // the child array (which is of String type) in this case will be:
+ // null: 1 0 0 0
+ // length: 0 1 1 1
+ // offset: 0 0 1 2
+ // and the array itself will be:
+ // null: 1 0
+ // length: 0 3
+ // offset: 0 1
+ //
+ // It's important that for the second element `[a, b, c]`, the offset starts from 1
+ // since otherwise we'd include the first null element from child array in the result.
+ offset += 1;
+ }
+ if (definitionLevel <= maxDefinitionLevel - 1) {
+ // Collection is null or one of its optional parents is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // Collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // Collection is defined and non-empty: find out how many top elements are there until the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
Review comment:
👍 Let me change it to `assembleStruct`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840755578
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +215,379 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
}
}
- state.advanceOffsetAndRowId(offset, rowId);
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ /**
+ * Reads a batch of repetition levels, definition levels and values into 'repLevels',
+ * 'defLevels' and 'values' respectively. The definition levels and values are read via
+ * 'defLevelsReader' and 'valueReader' respectively.
+ * <p>
+ * The related states such as row index, offset, number of rows left in the batch and page,
+ * are tracked by 'state'. The type-specific 'updater' is used to update or skip values.
+ */
+ public void readBatchRepeated(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchRepeatedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ /**
+ * Reads a batch of repetition levels, definition levels and integer values into 'repLevels',
+ * 'defLevels', 'values' and 'nulls' respectively. The definition levels and values are read via
+ * 'defLevelsReader' and 'valueReader' respectively.
+ * <p>
+ * The 'values' vector is used to hold non-null values, while 'nulls' vector is used to hold
+ * null values.
+ * <p>
+ * The related states such as row index, offset, number of rows left in the batch and page,
+ * are tracked by 'state'.
+ * <p>
+ * Unlike 'readBatchRepeated', this is used to decode dictionary indices in dictionary encoding.
+ */
+ public void readIntegersRepeated(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchRepeatedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchRepeatedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // Values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // The current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // This RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // Need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // The current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // The range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // Skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // Read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
+ currentCount -= n;
+ leftInBatch -= n;
+ leftInPage -= n;
+ }
+ }
+ } else {
+ // Not a top-level row: just read all the repetition levels in the block if the row
+ // should be included according to row indexes, else skip the rows.
+ if (!state.shouldSkip) {
+ repLevels.appendInts(valuesLeftInBlock, currentValue);
+ }
+ state.numBatchedDefLevels += valuesLeftInBlock;
+ leftInPage -= valuesLeftInBlock;
+ currentCount -= valuesLeftInBlock;
+ }
+ break;
+ case PACKED:
+ int i = 0;
+
+ for (; i < valuesLeftInBlock; i++) {
+ int currentValue = currentBuffer[currentBufferIdx + i];
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ break;
+ } else if (rowId < rangeStart) {
+ // This is a top-level row, therefore check if we should skip it with row indexes
+ // the row is before the current range, skip it
+ defLevelProcessor.skipValues(1);
+ } else if (rowId > rangeEnd) {
+ // The row is after the current range, move to the next range and compare again
+ state.nextRange();
+ break;
+ } else {
+ // The row is in the current range, decrement the row counter and read it
+ leftInBatch--;
+ repLevels.appendInt(0);
+ defLevelProcessor.readValues(1);
+ }
+ rowId++;
+ } else {
+ if (!state.shouldSkip) {
+ repLevels.appendInt(currentValue);
+ }
+ state.numBatchedDefLevels += 1;
+ }
+ }
+
+ leftInPage -= i;
+ currentCount -= i;
+ currentBufferIdx += i;
+ break;
+ }
+ }
+
+ // Process all the batched def levels
+ defLevelProcessor.finish();
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ private static class DefLevelProcessor {
+ private final VectorizedRleValuesReader reader;
+ private final ParquetReadState state;
+ private final WritableColumnVector defLevels;
+ private final WritableColumnVector values;
+ private final WritableColumnVector nulls;
+ private final boolean valuesReused;
+ private final VectorizedValuesReader valueReader;
+ private final ParquetVectorUpdater updater;
+
+ DefLevelProcessor(
+ VectorizedRleValuesReader reader,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ this.reader = reader;
+ this.state = state;
+ this.defLevels = defLevels;
+ this.values = values;
+ this.nulls = nulls;
+ this.valuesReused = valuesReused;
+ this.valueReader = valueReader;
+ this.updater = updater;
+ }
+
+ void readValues(int n) {
+ if (!state.shouldSkip) {
+ state.numBatchedDefLevels += n;
+ } else {
+ reader.skipValues(state.numBatchedDefLevels, state, valueReader, updater);
+ state.numBatchedDefLevels = n;
+ state.shouldSkip = false;
+ }
+ }
+
+ void skipValues(int n) {
+ if (state.shouldSkip) {
+ state.numBatchedDefLevels += n;
+ } else {
+ reader.readValues(state.numBatchedDefLevels, state, defLevels, values, nulls, valuesReused,
+ valueReader, updater);
+ state.numBatchedDefLevels = n;
+ state.shouldSkip = true;
+ }
+ }
+
+ void finish() {
+ if (state.numBatchedDefLevels > 0) {
+ if (state.shouldSkip) {
+ reader.skipValues(state.numBatchedDefLevels, state, valueReader, updater);
+ } else {
+ reader.readValues(state.numBatchedDefLevels, state, defLevels, values, nulls,
+ valuesReused, valueReader, updater);
+ }
+ state.numBatchedDefLevels = 0;
+ }
+ }
+ }
+
+ /**
+ * Read the next 'total' values (either null or non-null) from this definition level reader and
+ * 'valueReader'. The definition levels are read into 'defLevels'. If a value is not
+ * null, it is appended to 'values'. Otherwise, a null bit will be set in 'nulls'.
+ *
+ * This is only used when reading repeated values.
+ */
+ private void readValues(
+ int total,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ defLevels.reserveAdditional(total);
+ values.reserveAdditional(total);
+ if (!valuesReused) {
+ // 'nulls' is a separate column vector so we'll have to reserve it separately
+ nulls.reserveAdditional(total);
+ }
+
+ int n = total;
+ int initialValueOffset = state.valueOffset;
+ while (n > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+ int num = Math.min(n, this.currentCount);
+ switch (mode) {
+ case RLE:
+ if (currentValue == state.maxDefinitionLevel) {
+ updater.readValues(num, state.valueOffset, values, valueReader);
+ state.valueOffset += num;
+ } else {
+ // Only add null if this represents a null element, but not the case when a
+ // collection is null or empty.
+ nulls.putNulls(state.valueOffset, num);
+ state.valueOffset += num;
+ }
+ defLevels.putInts(state.levelOffset, num, currentValue);
Review comment:
👍 actually the comments here can also be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1086427714
Sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1035359009
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1021361169
@LuciferYang yes it is, let me rebase it to the latest master and address the comments by @klaus-xiong above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839086101
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
Review comment:
Yea `maxDefinitionLevel` and `maxRepetitionLevel` is for the collection itself here. For collection elements, the naming would be `maxElementDefinitionLevel` and `maxElementRepetitionLevel`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839068217
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
Review comment:
This is because `child.definitionLevel()` already incremented when the child is optional, so we need to negate it here. Still considering the example above. For optional elements:
```
RL DL
message schema { 0 0
optional group _1 (LIST) { 0 1
repeated group list { 1 2
optional binary element (UTF8); 1 3
}
}
}
```
The goal is to decide whether a value is an array element or not. Here `maxElementDefinitionLevel` is 3 - 1 = 2, and anything < 2 is not an array element.
```
```
RL DL
message schema { 0 0
optional group _1 (LIST) { 0 1
repeated group list { 1 2
required binary element (UTF8); 1 2
}
}
}
```
Similarly for required element, `maxElementDefinitionLevel` is 2.
I can change the variable name `maxElementDefinitionLevel` to something else if it's confusing :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r839065031
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel < maxElementDefinitionLevel) {
+ // This means the collection is null, and it is not an array element. In this case, we
+ // should increase offset to skip it when returning an array starting from the offset.
+ //
+ // For instance, considering an array of strings with 2 elements like the following:
+ // null, [a, b, c]
+ // the child array (which is of String type) in this case will be:
+ // null: 1 0 0 0
+ // length: 0 1 1 1
+ // offset: 0 0 1 2
+ // and the array itself will be:
+ // null: 1 0
+ // length: 0 3
+ // offset: 0 1
+ //
+ // It's important that for the second element `[a, b, c]`, the offset starts from 1
+ // since otherwise we'd include the first null element from child array in the result.
+ offset += 1;
+ }
+ if (definitionLevel <= maxDefinitionLevel - 1) {
+ // Collection is null or one of its optional parents is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // Collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // Collection is defined and non-empty: find out how many top elements are there until the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
+ int maxRepetitionLevel = column.repetitionLevel();
+ int maxDefinitionLevel = column.definitionLevel();
+
+ vector.reserve(definitionLevels.getElementsAppended());
+
+ int rowId = 0;
+ boolean hasRepetitionLevels = repetitionLevels.getElementsAppended() > 0;
+ for (int i = 0; i < definitionLevels.getElementsAppended(); i++) {
+ // If repetition level > maxRepetitionLevel, the value is a nested element (e.g., an array
+ // element in struct<array<int>>), and we should skip the definition level since it doesn't
+ // represent with the struct.
+ if (!hasRepetitionLevels || repetitionLevels.getInt(i) <= maxRepetitionLevel) {
+ if (definitionLevels.getInt(i) <= maxDefinitionLevel - 1) {
+ // Struct is null
+ vector.putNull(rowId);
+ rowId++;
+ } else if (definitionLevels.getInt(i) >= maxDefinitionLevel) {
+ vector.putNotNull(rowId);
+ rowId++;
+ }
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private static WritableColumnVector allocateLevelsVector(int capacity, MemoryMode memoryMode) {
+ switch (memoryMode) {
+ case ON_HEAP:
+ return new OnHeapColumnVector(capacity, DataTypes.IntegerType);
+ case OFF_HEAP:
+ return new OffHeapColumnVector(capacity, DataTypes.IntegerType);
+ default:
+ throw new IllegalArgumentException("Unknown memory mode: " + memoryMode);
+ }
+ }
+
+ /**
+ * For a collection (i.e., array or map) element at index 'idx', returns the starting index of
+ * the next collection after it.
+ *
+ * @param maxRepetitionLevel the maximum repetition level for the elements in this collection
+ * @param idx the index of this collection in the Parquet column
+ * @return the starting index of the next collection
+ */
+ private int getNextCollectionStart(int maxRepetitionLevel, int idx) {
+ idx += 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
Review comment:
The `maxRepetitionLevel` is the repetition level of the array itself. Drawing the previous example:
```
RL DL
message schema { 0 0
optional group _1 (LIST) { 0 1
repeated group list { 1 2
optional binary element (UTF8); 1 3
}
}
}
```
the `maxRepetitionLevel` is 0 in this case. Therefore, for an array of `[[0, 1, 2], [3, 4, 5]]`, it's repetition levels will be: `[0, 1, 1, 0, 1, 1]`. The if condition returns true when `repetitionLevels.getInt(idx)` is equal to 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840730069
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -187,6 +188,30 @@ object ParquetUtils {
}
}
+ /**
+ * Whether columnar read is supported for the input `schema`.
+ */
+ def isBatchReadSupportedForSchema(sqlConf: SQLConf, schema: StructType): Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ schema.forall(f => isBatchReadSupported(sqlConf, f.dataType))
+
+ def isBatchReadSupported(sqlConf: SQLConf, dt: DataType): Boolean = dt match {
+ case _: AtomicType =>
+ true
+ case at: ArrayType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, at.elementType)
+ case mt: MapType =>
+ sqlConf.parquetVectorizedReaderNestedColumnEnabled &&
+ isBatchReadSupported(sqlConf, mt.keyType) &&
Review comment:
let me check all the places with 4 space indentation ... I need to fix my IntelliJ settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1086094238
@dongjoon-hyun I think it's unrelated to this PR - looks like it's due to https://github.com/apache/spark/pull/36012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1085432235
Hi, @sunchao . After rebasing and force-pushing, the PR branch is broken. Could you take a look?
```
[error] /home/runner/work/spark/spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala:426:22: not enough arguments for constructor TaskContextImpl: (stageId: Int, stageAttemptNumber: Int, partitionId: Int, taskAttemptId: Long, attemptNumber: Int, numPartitions: Int, taskMemoryManager: org.apache.spark.memory.TaskMemoryManager, localProperties: java.util.Properties, metricsSystem: org.apache.spark.metrics.MetricsSystem, taskMetrics: org.apache.spark.executor.TaskMetrics, cpus: Int, resources: Map[String,org.apache.spark.resource.ResourceInformation])org.apache.spark.TaskContextImpl.
(https://github.com/sunchao/spark/runs/5778352461?check_suite_focus=true#step:9:600)
[error] Unspecified value parameter metricsSystem.
(https://github.com/sunchao/spark/runs/5778352461?check_suite_focus=true#step:9:601)
[error] val context1 = new TaskContextImpl(0, 0, 0, 0, 0, null, null, null, cpus = 0)
(https://github.com/sunchao/spark/runs/5778352461?check_suite_focus=true#step:9:602)
[error] ^
(https://github.com/sunchao/spark/runs/5778352461?check_suite_focus=true#step:9:603)
[error] one error found
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r840675154
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +215,379 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
}
}
- state.advanceOffsetAndRowId(offset, rowId);
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ /**
+ * Reads a batch of repetition levels, definition levels and values into 'repLevels',
+ * 'defLevels' and 'values' respectively. The definition levels and values are read via
+ * 'defLevelsReader' and 'valueReader' respectively.
+ * <p>
+ * The related states such as row index, offset, number of rows left in the batch and page,
+ * are tracked by 'state'. The type-specific 'updater' is used to update or skip values.
+ */
+ public void readBatchRepeated(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchRepeatedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ /**
+ * Reads a batch of repetition levels, definition levels and integer values into 'repLevels',
+ * 'defLevels', 'values' and 'nulls' respectively. The definition levels and values are read via
+ * 'defLevelsReader' and 'valueReader' respectively.
+ * <p>
+ * The 'values' vector is used to hold non-null values, while 'nulls' vector is used to hold
+ * null values.
+ * <p>
+ * The related states such as row index, offset, number of rows left in the batch and page,
+ * are tracked by 'state'.
+ * <p>
+ * Unlike 'readBatchRepeated', this is used to decode dictionary indices in dictionary encoding.
+ */
+ public void readIntegersRepeated(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchRepeatedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchRepeatedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // Values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // The current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // This RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // Need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // The current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // The range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // Skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // Read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
+ currentCount -= n;
+ leftInBatch -= n;
+ leftInPage -= n;
+ }
+ }
+ } else {
+ // Not a top-level row: just read all the repetition levels in the block if the row
+ // should be included according to row indexes, else skip the rows.
+ if (!state.shouldSkip) {
+ repLevels.appendInts(valuesLeftInBlock, currentValue);
+ }
+ state.numBatchedDefLevels += valuesLeftInBlock;
+ leftInPage -= valuesLeftInBlock;
+ currentCount -= valuesLeftInBlock;
+ }
+ break;
+ case PACKED:
+ int i = 0;
+
+ for (; i < valuesLeftInBlock; i++) {
+ int currentValue = currentBuffer[currentBufferIdx + i];
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ break;
+ } else if (rowId < rangeStart) {
+ // This is a top-level row, therefore check if we should skip it with row indexes
+ // the row is before the current range, skip it
+ defLevelProcessor.skipValues(1);
+ } else if (rowId > rangeEnd) {
+ // The row is after the current range, move to the next range and compare again
+ state.nextRange();
+ break;
+ } else {
+ // The row is in the current range, decrement the row counter and read it
+ leftInBatch--;
+ repLevels.appendInt(0);
+ defLevelProcessor.readValues(1);
+ }
+ rowId++;
+ } else {
+ if (!state.shouldSkip) {
+ repLevels.appendInt(currentValue);
+ }
+ state.numBatchedDefLevels += 1;
+ }
+ }
+
+ leftInPage -= i;
+ currentCount -= i;
+ currentBufferIdx += i;
+ break;
+ }
+ }
+
+ // Process all the batched def levels
+ defLevelProcessor.finish();
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ private static class DefLevelProcessor {
+ private final VectorizedRleValuesReader reader;
+ private final ParquetReadState state;
+ private final WritableColumnVector defLevels;
+ private final WritableColumnVector values;
+ private final WritableColumnVector nulls;
+ private final boolean valuesReused;
+ private final VectorizedValuesReader valueReader;
+ private final ParquetVectorUpdater updater;
+
+ DefLevelProcessor(
+ VectorizedRleValuesReader reader,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ this.reader = reader;
+ this.state = state;
+ this.defLevels = defLevels;
+ this.values = values;
+ this.nulls = nulls;
+ this.valuesReused = valuesReused;
+ this.valueReader = valueReader;
+ this.updater = updater;
+ }
+
+ void readValues(int n) {
+ if (!state.shouldSkip) {
+ state.numBatchedDefLevels += n;
+ } else {
+ reader.skipValues(state.numBatchedDefLevels, state, valueReader, updater);
+ state.numBatchedDefLevels = n;
+ state.shouldSkip = false;
+ }
+ }
+
+ void skipValues(int n) {
+ if (state.shouldSkip) {
+ state.numBatchedDefLevels += n;
+ } else {
+ reader.readValues(state.numBatchedDefLevels, state, defLevels, values, nulls, valuesReused,
+ valueReader, updater);
+ state.numBatchedDefLevels = n;
+ state.shouldSkip = true;
+ }
+ }
+
+ void finish() {
+ if (state.numBatchedDefLevels > 0) {
+ if (state.shouldSkip) {
+ reader.skipValues(state.numBatchedDefLevels, state, valueReader, updater);
+ } else {
+ reader.readValues(state.numBatchedDefLevels, state, defLevels, values, nulls,
+ valuesReused, valueReader, updater);
+ }
+ state.numBatchedDefLevels = 0;
+ }
+ }
+ }
+
+ /**
+ * Read the next 'total' values (either null or non-null) from this definition level reader and
+ * 'valueReader'. The definition levels are read into 'defLevels'. If a value is not
+ * null, it is appended to 'values'. Otherwise, a null bit will be set in 'nulls'.
+ *
+ * This is only used when reading repeated values.
+ */
+ private void readValues(
+ int total,
+ ParquetReadState state,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ defLevels.reserveAdditional(total);
+ values.reserveAdditional(total);
+ if (!valuesReused) {
+ // 'nulls' is a separate column vector so we'll have to reserve it separately
+ nulls.reserveAdditional(total);
+ }
+
+ int n = total;
+ int initialValueOffset = state.valueOffset;
+ while (n > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+ int num = Math.min(n, this.currentCount);
+ switch (mode) {
+ case RLE:
+ if (currentValue == state.maxDefinitionLevel) {
+ updater.readValues(num, state.valueOffset, values, valueReader);
+ state.valueOffset += num;
+ } else {
+ // Only add null if this represents a null element, but not the case when a
+ // collection is null or empty.
+ nulls.putNulls(state.valueOffset, num);
+ state.valueOffset += num;
+ }
+ defLevels.putInts(state.levelOffset, num, currentValue);
+ break;
+ case PACKED:
Review comment:
line 565 ~ 575 looks same as line 247 ~ 257 except for the comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r803329816
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
Review comment:
Should we lazy initializing `children` after line 86 and use `column.children()` to specify the initialCapacity of `ArrayList`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-1021361169
@LuciferYang yes it is, let me rebase it to the latest master and address the comments by @klaus-xiong above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r771852716
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
##########
@@ -169,8 +170,8 @@ class ParquetFileFormat
override def supportBatch(sparkSession: SparkSession, schema: StructType): Boolean = {
val conf = sparkSession.sessionState.conf
conf.parquetVectorizedReaderEnabled && conf.wholeStageEnabled &&
- schema.length <= conf.wholeStageMaxNumFields &&
- schema.forall(_.dataType.isInstanceOf[AtomicType])
+ ParquetFileFormat.isBatchReadSupported(conf, schema) &&
Review comment:
Good catch! I think it should be:
```scala
schema.forall(f => ParquetFileFormat.isBatchReadSupported(conf, f.dataType))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998352669
**[Test build #146411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146411/testReport)** for PR 34659 at commit [`aff8504`](https://github.com/apache/spark/commit/aff8504d4130095ee1ecfdab3f3d2d9110d40400).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998400058
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50886/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r772409538
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -350,6 +355,14 @@ public WritableColumnVector reserveDictionaryIds(int capacity) {
*/
public abstract void putArray(int rowId, int offset, int length);
+ /**
+ * Puts a new non-null struct at 'rowId' of this vector, which is backed by elements at
+ * 'offset' of child vectors.
+ *
+ * NOTE: this MUST be called after new elements are appended to child vectors of a struct vector.
+ */
+ public abstract void putStruct(int rowId, int offset);
Review comment:
yea `WritableColumnVector` is an internal API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-998398555
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50886/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-999823625
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50965/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760636401
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -227,30 +230,340 @@ private void readBatchInternal(
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
- } else {
- nulls.putNull(offset + i);
+ int v = currentBuffer[currentBufferIdx++];
+ if (v == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
+ } else if (!state.isRequired && v == state.maxDefinitionLevel - 1) {
+ // only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, v);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // the current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // this RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
Review comment:
it shouldn't since we don't add anything to the batch here (we are skipping rows).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34659:
URL: https://github.com/apache/spark/pull/34659#issuecomment-984181954
**[Test build #145831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145831/testReport)** for PR 34659 at commit [`7b6e083`](https://github.com/apache/spark/commit/7b6e083833c807e44b76bc321d32e9c5857e648f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r760522527
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // this can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parent is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
+ //
+ // `i` is the index over all leaf elements of this array, while `offset` is the index over
+ // all top-level elements of this array.
+ int rowId = 0;
+ for (int i = 0, offset = 0; i < definitionLevels.getElementsAppended();
+ i = getNextCollectionStart(maxElementRepetitionLevel, i)) {
+ vector.reserve(rowId + 1);
+ int definitionLevel = definitionLevels.getInt(i);
+ if (definitionLevel == maxDefinitionLevel - 1) {
+ // the collection is null
+ vector.putNull(rowId++);
+ } else if (definitionLevel == maxDefinitionLevel) {
+ // collection is defined but empty
+ vector.putNotNull(rowId);
+ vector.putArray(rowId, offset, 0);
+ rowId++;
+ } else if (definitionLevel > maxDefinitionLevel) {
+ // collection is defined and non-empty: find out how many top element there is till the
+ // start of the next array.
+ vector.putNotNull(rowId);
+ int length = getCollectionSize(maxElementRepetitionLevel, i + 1);
+ vector.putArray(rowId, offset, length);
+ offset += length;
+ rowId++;
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private void calculateStructOffsets() {
+ int maxRepetitionLevel = column.repetitionLevel();
+ int maxDefinitionLevel = column.definitionLevel();
+
+ vector.reserve(definitionLevels.getElementsAppended());
+
+ int rowId = 0;
+ int nonnullRowId = 0;
+ boolean hasRepetitionLevels = repetitionLevels.getElementsAppended() > 0;
+ for (int i = 0; i < definitionLevels.getElementsAppended(); i++) {
+ // if repetition level > maxRepetitionLevel, the value is a nested element (e.g., an array
+ // element in struct<array<int>>), and we should skip the definition level since it doesn't
+ // represent with the struct.
+ if (!hasRepetitionLevels || repetitionLevels.getInt(i) <= maxRepetitionLevel) {
+ if (definitionLevels.getInt(i) == maxDefinitionLevel - 1) {
+ // the struct is null
+ vector.putNull(rowId);
+ rowId++;
+ } else if (definitionLevels.getInt(i) >= maxDefinitionLevel) {
+ vector.putNotNull(rowId);
+ vector.putStruct(rowId, nonnullRowId);
+ rowId++;
+ nonnullRowId++;
+ }
+ }
+ }
+ vector.addElementsAppended(rowId);
+ }
+
+ private static WritableColumnVector allocateLevelsVector(int capacity, MemoryMode memoryMode) {
+ switch (memoryMode) {
+ case ON_HEAP:
+ return new OnHeapColumnVector(capacity, DataTypes.IntegerType);
+ case OFF_HEAP:
+ return new OffHeapColumnVector(capacity, DataTypes.IntegerType);
+ default:
+ throw new IllegalArgumentException("Unknown memory mode: " + memoryMode);
+ }
+ }
+
+ private int getNextCollectionStart(int maxRepetitionLevel, int elementIndex) {
+ int idx = elementIndex + 1;
+ for (; idx < repetitionLevels.getElementsAppended(); idx++) {
+ if (repetitionLevels.getInt(idx) <= maxRepetitionLevel) {
+ break;
+ }
+ }
+ return idx;
+ }
+
+ private int getCollectionSize(int maxRepetitionLevel, int idx) {
+ int size = 1;
Review comment:
Yes, it is because the collection is non-empty, and `idx` is pointing to the next element after the first element in the collection.
> Depending on the idx, we could double count size or skip elements
Hmm why this will happen? I think it's better to calculate the final size in the method instead of having to add +1 in every call site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r834884862
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -246,25 +249,25 @@ private void initBatch(
}
}
+ WritableColumnVector[] vectors;
if (memMode == MemoryMode.OFF_HEAP) {
- columnVectors = OffHeapColumnVector.allocateColumns(capacity, batchSchema);
+ vectors = OffHeapColumnVector.allocateColumns(capacity, batchSchema);
} else {
- columnVectors = OnHeapColumnVector.allocateColumns(capacity, batchSchema);
+ vectors = OnHeapColumnVector.allocateColumns(capacity, batchSchema);
+ }
+ columnarBatch = new ColumnarBatch(vectors);
+
+ columnVectors = new ParquetColumnVector[sparkSchema.fields().length];
+ for (int i = 0; i < columnVectors.length; i++) {
+ columnVectors[i] = new ParquetColumnVector(parquetColumn.children().apply(i),
Review comment:
This looks a bit weird, `parquetColumn.children().apply(i)`. Actually it is the `i` column in required schema, right?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -157,7 +160,11 @@ private boolean isLazyDecodingSupported(PrimitiveType.PrimitiveTypeName typeName
/**
* Reads `total` values from this columnReader into column.
Review comment:
```suggestion
* Reads `total` rows from this columnReader into column.
```
is it correct?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -214,24 +231,32 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
boolean needTransform = castLongToInt || isUnsignedInt32 || isUnsignedInt64;
column.setDictionary(new ParquetDictionary(dictionary, needTransform));
} else {
- updater.decodeDictionaryIds(readState.offset - startOffset, startOffset, column,
+ updater.decodeDictionaryIds(readState.valueOffset - startOffset, startOffset, column,
dictionaryIds, dictionary);
}
} else {
- if (column.hasDictionary() && readState.offset != 0) {
+ if (column.hasDictionary() && readState.valueOffset != 0) {
// This batch already has dictionary encoded values but this new page is not. The batch
// does not support a mix of dictionary and not so we will decode the dictionary.
- updater.decodeDictionaryIds(readState.offset, 0, column, dictionaryIds, dictionary);
+ updater.decodeDictionaryIds(readState.valueOffset, 0, column, dictionaryIds, dictionary);
}
column.setDictionary(null);
VectorizedValuesReader valuesReader = (VectorizedValuesReader) dataColumn;
- defColumn.readBatch(readState, column, valuesReader, updater);
+ if (readState.maxRepetitionLevel == 0) {
+ defColumn.readBatch(readState, column, definitionLevels, valuesReader, updater);
+ } else {
+ repColumn.readBatchNested(readState, repetitionLevels, defColumn, definitionLevels,
+ column, valuesReader, updater);
+ }
}
}
}
private int readPage() {
DataPage page = pageReader.readPage();
+ if (page == null) {
+ return -1;
+ }
Review comment:
Is this related to nested change?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
Review comment:
Seems this is used to read repetition level column only? Could you add a comment here too?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
Review comment:
nit: this looks more like `readBatchForRepeated`? Because looks like `readBatch` also reads nested but not repeated column?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
Review comment:
`Keep reading repetition level values ...` means this only reads repetition level values? Not data?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ VectorizedValuesReader valueReader) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, nulls, false,
+ valueReader, new ParquetVectorUpdaterFactory.IntegerUpdater());
+ }
+
+ /**
+ * Keep reading repetition level values from the page until either: 1) we've read enough
+ * top-level rows to fill the current batch, or 2) we've drained the data page completely.
+ *
+ * @param valuesReused whether 'values' vector is reused for 'nulls'
+ */
+ public void readBatchNestedInternal(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ WritableColumnVector nulls,
+ boolean valuesReused,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+
+ int leftInBatch = state.rowsToReadInBatch;
+ int leftInPage = state.valuesToReadInPage;
+ long rowId = state.rowId;
+
+ DefLevelProcessor defLevelProcessor = new DefLevelProcessor(defLevelsReader, state, defLevels,
+ values, nulls, valuesReused, valueReader, updater);
+
+ while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
+ if (currentCount == 0 && !readNextGroup()) break;
+
+ // Values to read in the current RLE/PACKED block, must be <= what's left in the page
+ int valuesLeftInBlock = Math.min(leftInPage, currentCount);
+
+ // The current row range start and end
+ long rangeStart = state.currentRangeStart();
+ long rangeEnd = state.currentRangeEnd();
+
+ switch (mode) {
+ case RLE:
+ // This RLE block is consist of top-level rows, so we'll need to check
+ // if the rows should be skipped according to row indexes.
+ if (currentValue == 0) {
+ if (leftInBatch == 0) {
+ state.lastListCompleted = true;
+ } else {
+ // # of rows to read in the block, must be <= what's left in the current batch
+ int n = Math.min(leftInBatch, valuesLeftInBlock);
+
+ if (rowId + n < rangeStart) {
+ // Need to skip all rows in [rowId, rowId + n)
+ defLevelProcessor.skipValues(n);
+ rowId += n;
+ currentCount -= n;
+ leftInPage -= n;
+ } else if (rowId > rangeEnd) {
+ // The current row index already beyond the current range: move to the next range
+ // and repeat
+ state.nextRange();
+ } else {
+ // The range [rowId, rowId + n) overlaps with the current row range
+ long start = Math.max(rangeStart, rowId);
+ long end = Math.min(rangeEnd, rowId + n - 1);
+
+ // Skip the rows in [rowId, start)
+ int toSkip = (int) (start - rowId);
+ if (toSkip > 0) {
+ defLevelProcessor.skipValues(toSkip);
+ rowId += toSkip;
+ currentCount -= toSkip;
+ leftInPage -= toSkip;
+ }
+
+ // Read the rows in [start, end]
+ n = (int) (end - start + 1);
+
+ if (n > 0) {
+ repLevels.appendInts(n, 0);
+ defLevelProcessor.readValues(n);
+ }
+
+ rowId += n;
+ currentCount -= n;
+ leftInBatch -= n;
+ leftInPage -= n;
+ }
+ }
+ } else {
+ // Not a top-level row: just read all the repetition levels in the block if the row
+ // should be included according to row indexes, else skip the rows.
+ if (!state.shouldSkip) {
+ repLevels.appendInts(valuesLeftInBlock, currentValue);
+ }
+ state.numBatchedDefLevels += valuesLeftInBlock;
Review comment:
Don't we need to read definition level values `valuesLeftInBlock`?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -210,48 +213,358 @@ private void readBatchInternal(
} else if (rowId > rangeEnd) {
state.nextRange();
} else {
- // the range [rowId, rowId + n) overlaps with the current row range in state
+ // The range [rowId, rowId + n) overlaps with the current row range in state
long start = Math.max(rangeStart, rowId);
long end = Math.min(rangeEnd, rowId + n - 1);
- // skip the part [rowId, start)
+ // Skip the part [rowId, start)
int toSkip = (int) (start - rowId);
if (toSkip > 0) {
skipValues(toSkip, state, valueReader, updater);
rowId += toSkip;
leftInPage -= toSkip;
}
- // read the part [start, end]
+ // Read the part [start, end]
n = (int) (end - start + 1);
switch (mode) {
case RLE:
if (currentValue == state.maxDefinitionLevel) {
- updater.readValues(n, offset, values, valueReader);
- } else {
- nulls.putNulls(offset, n);
+ updater.readValues(n, state.valueOffset, values, valueReader);
+ state.valueOffset += n;
+ } else if (!state.isRequired && currentValue == state.maxDefinitionLevel - 1) {
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNulls(state.valueOffset, n);
+ state.valueOffset += n;
}
+ defLevels.putInts(state.levelOffset, n, currentValue);
break;
case PACKED:
for (int i = 0; i < n; ++i) {
- if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) {
- updater.readValue(offset + i, values, valueReader);
+ int value = currentBuffer[currentBufferIdx++];
+ if (value == state.maxDefinitionLevel) {
+ updater.readValue(state.valueOffset++, values, valueReader);
} else {
- nulls.putNull(offset + i);
+ // Only add null if this represents a null element, but not for the case where a
+ // struct itself is null
+ nulls.putNull(state.valueOffset++);
}
+ defLevels.putInt(state.levelOffset + i, value);
}
break;
}
- offset += n;
+ state.levelOffset += n;
leftInBatch -= n;
rowId += n;
leftInPage -= n;
currentCount -= n;
+ defLevels.addElementsAppended(n);
+ }
+ }
+
+ state.rowsToReadInBatch = leftInBatch;
+ state.valuesToReadInPage = leftInPage;
+ state.rowId = rowId;
+ }
+
+ public void readBatchNested(
+ ParquetReadState state,
+ WritableColumnVector repLevels,
+ VectorizedRleValuesReader defLevelsReader,
+ WritableColumnVector defLevels,
+ WritableColumnVector values,
+ VectorizedValuesReader valueReader,
+ ParquetVectorUpdater updater) {
+ readBatchNestedInternal(state, repLevels, defLevelsReader, defLevels, values, values, true,
+ valueReader, updater);
+ }
+
+ public void readIntegersNested(
Review comment:
What difference between this and `readBatchNested`? Maybe add a comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34659:
URL: https://github.com/apache/spark/pull/34659#discussion_r834905245
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import com.google.common.base.Preconditions;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * Contains necessary information representing a Parquet column, either of primitive or nested type.
+ */
+final class ParquetColumnVector {
+ private final ParquetColumn column;
+ private final List<ParquetColumnVector> children;
+ private final WritableColumnVector vector;
+
+ /**
+ * Repetition & Definition levels
+ * These are allocated only for leaf columns; for non-leaf columns, they simply maintain
+ * references to that of the former.
+ */
+ private WritableColumnVector repetitionLevels;
+ private WritableColumnVector definitionLevels;
+
+ /** Whether this column is primitive (i.e., leaf column) */
+ private final boolean isPrimitive;
+
+ /** Reader for this column - only set if 'isPrimitive' is true */
+ private VectorizedColumnReader columnReader;
+
+ ParquetColumnVector(
+ ParquetColumn column,
+ WritableColumnVector vector,
+ int capacity,
+ MemoryMode memoryMode,
+ Set<ParquetColumn> missingColumns) {
+
+ DataType sparkType = column.sparkType();
+ if (!sparkType.sameType(vector.dataType())) {
+ throw new IllegalArgumentException("Spark type: " + sparkType +
+ " doesn't match the type: " + vector.dataType() + " in column vector");
+ }
+
+ this.column = column;
+ this.vector = vector;
+ this.children = new ArrayList<>();
+ this.isPrimitive = column.isPrimitive();
+
+ if (missingColumns.contains(column)) {
+ vector.setAllNull();
+ return;
+ }
+
+ if (isPrimitive) {
+ // TODO: avoid allocating these if not necessary, for instance, the node is of top-level
+ // and is not repeated, or the node is not top-level but its max repetition level is 0.
+ repetitionLevels = allocateLevelsVector(capacity, memoryMode);
+ definitionLevels = allocateLevelsVector(capacity, memoryMode);
+ } else {
+ Preconditions.checkArgument(column.children().size() == vector.getNumChildren());
+ for (int i = 0; i < column.children().size(); i++) {
+ ParquetColumnVector childCv = new ParquetColumnVector(column.children().apply(i),
+ vector.getChild(i), capacity, memoryMode, missingColumns);
+ children.add(childCv);
+
+ // Only use levels from non-missing child, this can happen if only some but not all
+ // fields of a struct are missing.
+ if (!childCv.vector.isAllNull()) {
+ this.repetitionLevels = childCv.repetitionLevels;
+ this.definitionLevels = childCv.definitionLevels;
+ }
+ }
+
+ // This can happen if all the fields of a struct are missing, in which case we should mark
+ // the struct itself as a missing column
+ if (repetitionLevels == null) {
+ vector.setAllNull();
+ }
+ }
+ }
+
+ /**
+ * Returns all the children of this column.
+ */
+ List<ParquetColumnVector> getChildren() {
+ return children;
+ }
+
+ /**
+ * Returns all the leaf columns in depth-first order.
+ */
+ List<ParquetColumnVector> getLeaves() {
+ List<ParquetColumnVector> result = new ArrayList<>();
+ getLeavesHelper(this, result);
+ return result;
+ }
+
+ private static void getLeavesHelper(ParquetColumnVector vector, List<ParquetColumnVector> coll) {
+ if (vector.isPrimitive) {
+ coll.add(vector);
+ } else {
+ for (ParquetColumnVector child : vector.children) {
+ getLeavesHelper(child, coll);
+ }
+ }
+ }
+
+ /**
+ * Assembles this column and calculate collection offsets recursively.
+ * This is a no-op for primitive columns.
+ */
+ void assemble() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ DataType type = column.sparkType();
+ if (type instanceof ArrayType || type instanceof MapType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateCollectionOffsets();
+ } else if (type instanceof StructType) {
+ for (ParquetColumnVector child : children) {
+ child.assemble();
+ }
+ calculateStructOffsets();
+ }
+ }
+
+ /**
+ * Resets this Parquet column vector, which includes resetting all the writable column vectors
+ * (used to store values, definition levels, and repetition levels) for this and all its children.
+ */
+ void reset() {
+ // nothing to do if the column itself is missing
+ if (vector.isAllNull()) return;
+
+ vector.reset();
+ repetitionLevels.reset();
+ definitionLevels.reset();
+ for (ParquetColumnVector child : children) {
+ child.reset();
+ }
+ }
+
+ /**
+ * Returns the {@link ParquetColumn} of this column vector.
+ */
+ ParquetColumn getColumn() {
+ return this.column;
+ }
+
+ /**
+ * Returns the writable column vector used to store values.
+ */
+ WritableColumnVector getValueVector() {
+ return this.vector;
+ }
+
+ /**
+ * Returns the writable column vector used to store repetition levels.
+ */
+ WritableColumnVector getRepetitionLevelVector() {
+ return this.repetitionLevels;
+ }
+
+ /**
+ * Returns the writable column vector used to store definition levels.
+ */
+ WritableColumnVector getDefinitionLevelVector() {
+ return this.definitionLevels;
+ }
+
+ /**
+ * Returns the column reader for reading a Parquet column.
+ */
+ VectorizedColumnReader getColumnReader() {
+ return this.columnReader;
+ }
+
+ /**
+ * Sets the column vector to 'reader'. Note this can only be called on a primitive Parquet
+ * column.
+ */
+ void setColumnReader(VectorizedColumnReader reader) {
+ if (!isPrimitive) {
+ throw new IllegalStateException("Can't set reader for non-primitive column");
+ }
+ this.columnReader = reader;
+ }
+
+ private void calculateCollectionOffsets() {
+ int maxDefinitionLevel = column.definitionLevel();
+ int maxElementRepetitionLevel = column.repetitionLevel();
+
+ ParquetColumn child = column.children().apply(0);
+ int maxElementDefinitionLevel = child.definitionLevel();
+ if (!child.required()) maxElementDefinitionLevel -= 1;
+
+ // There are 4 cases when calculating definition levels:
+ // 1. definitionLevel == maxDefinitionLevel
+ // ==> value is defined and not null
+ // 2. definitionLevel == maxDefinitionLevel - 1
+ // ==> value is null
+ // 3. definitionLevel < maxDefinitionLevel - 1
+ // ==> value doesn't exist since one of its optional parents is null
+ // 4. definitionLevel > maxDefinitionLevel
+ // ==> value is a nested element within an array or map
Review comment:
Any example for case 4? You mean the `maxDefinitionLevel` is the definition level of the array? Then isn't the `maxDefinitionLevel` should be `array's definitionLevel + 1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org