You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/02 22:13:17 UTC
[GitHub] [spark] parthchandra opened a new pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
parthchandra opened a new pull request #34471:
URL: https://github.com/apache/spark/pull/34471
### What changes were proposed in this pull request?
Implements a vectorized version of the parquet reader for DELTA_BINARY_PACKED encoding
This PR includes a previous PR for this issue which passed the read request thru to the parquet implementation and which was not vectorized. The current PR builds on top of that PR (hence both are included).
### Why are the changes needed?
Currently Spark throws an exception when reading data with these encodings if vectorized reader is enabled
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Additional unit tests for the encoding for both long and integer types (mirroring the unit tests in the Parquet implementation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r779406568
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -76,6 +79,7 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
saveAsCsvTable(testDf, dir.getCanonicalPath + "/csv")
saveAsJsonTable(testDf, dir.getCanonicalPath + "/json")
saveAsParquetTable(testDf, dir.getCanonicalPath + "/parquet")
+ saveAsParquetV2Table(testDf, dir.getCanonicalPath + "/parquetV2")
Review comment:
Maybe we should update the benchmark-result of `DataSourceReadBenchmark` @parthchandra parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959814220
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49350/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958302148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958531680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958446671
**[Test build #144864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144864/testReport)** for PR 34471 at commit [`10659d3`](https://github.com/apache/spark/commit/10659d3bc9c1f3b3b9f8c29000f9f75e95281777).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958589503
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49334/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966714987
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49592/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966807834
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972578844
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49839/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972551792
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49839/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r768213546
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -147,6 +159,16 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
}
}
+ sqlBenchmark.addCase("SQL Parquet Vectorized (Delta Binary)") { _ =>
Review comment:
I guess I didn't understand the implication when you suggested I need to rebase!
It looks like the Boolean type breaks vectorized reading. The boolean column gets encoded as RLE but the vectorized RLE reader does not support Boolean.
```
[info] Caused by: java.lang.UnsupportedOperationException: only readInts is valid.
[info] at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBooleans(VectorizedRleValuesReader.java:368)
```
I guess I could fix this. Would you recommend a different PR, or should I make the change in this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r750611564
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @SuppressWarnings("unused")
Review comment:
nit: do we need this annotation?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
Review comment:
nit: extra blank line.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
+ }
+
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
Review comment:
in that case can we put this together with the follow-up PR?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ return deltaByteArrayReader.readBytes();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ for (int i = 0; i < total; i++) {
+ Binary binary = deltaByteArrayReader.readBytes();
+ ByteBuffer buffer = binary.toByteBuffer();
+ if (buffer.hasArray()) {
+ c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
+ binary.length());
+ } else {
+ byte[] bytes = new byte[binary.length()];
+ buffer.get(bytes);
+ c.putByteArray(rowId + i, bytes);
+ }
+ }
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ for(int i =0; i < total; i++) {
Review comment:
nit: space after `=`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedReaderBase.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * Base class for implementations of VectorizedValuesReader. Mainly to avoid duplication
+ * of methods that are not supported by concrete implementations
+ */
+public class VectorizedReaderBase extends ValuesReader implements VectorizedValuesReader {
Review comment:
👍 it'd be good to have others like `VectorizedRleValuesReader` to extend this too, but can be done separately.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
##########
@@ -168,6 +169,16 @@ private[sql] trait ParquetTest extends FileBasedDataSourceTest {
withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true")(code)
}
+ def withAllParquetWriters(code: => Unit): Unit = {
+ ParquetOutputFormat.WRITER_VERSION -> ParquetProperties.WriterVersion.PARQUET_2_0.toString
Review comment:
what is this line for?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ BytePackerForLong packer = Packer.LITTLE_ENDIAN.newBytePackerForLong(
+ bitWidths[currentMiniBlock]);
+ for (int j = 0; j < miniBlockSizeInValues; j += 8) {
+ ByteBuffer buffer = in.slice(packer.getBitWidth());
+ packer.unpack8Values(buffer, buffer.position(), unpackedValuesBuffer, j);
Review comment:
I see, thanks for the reference!
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
Review comment:
We also increment `valuesRead` in the same loop though, and we have `valuesRead < remaining` in the loop condition. This doesn't look right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958404969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959699641
Jenkins, retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r744039103
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
Review comment:
Could you put this at the beginning before `int remaining = total;`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r744037047
##########
File path: .gitignore
##########
@@ -33,6 +33,7 @@ R/pkg/tests/fulltests/Rplots.pdf
build/*.jar
build/apache-maven*
build/scala*
+build/zinc*
Review comment:
+1 with @sunchao 's comment. Please remove this from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966738491
**[Test build #145125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145125/testReport)** for PR 34471 at commit [`a88c721`](https://github.com/apache/spark/commit/a88c72169e7c3e624db1303ab5db1bfac253eb37).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966738491
**[Test build #145125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145125/testReport)** for PR 34471 at commit [`a88c721`](https://github.com/apache/spark/commit/a88c72169e7c3e624db1303ab5db1bfac253eb37).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r767025178
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
Review comment:
removed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
Review comment:
aligned it properly.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putByte(r, (byte) v));
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putShort(r, (short) v));
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putInt(r, (int) v));
+ }
+
+ // Based on VectorizedPlainValuesReader.readIntegersWithRebase
+ @Override
+ public final void readIntegersWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putInt(r, RebaseDateTime.rebaseJulianToGregorianDays((int) v));
+ }
+ } else {
+ w.putInt(r, (int) v);
+ }
+ });
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putLong(r, Integer.toUnsignedLong((int) v));
+ });
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putByteArray(r, new BigInteger(Long.toUnsignedString(v)).toByteArray());
+ });
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public final void readLongsWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
Review comment:
Yes, it should. Thank you for catching this!
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -147,6 +159,16 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
}
}
+ sqlBenchmark.addCase("SQL Parquet Vectorized (Delta Binary)") { _ =>
Review comment:
Done
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -16,14 +16,22 @@
*/
package org.apache.spark.sql.execution.datasources.parquet
+import java.math.BigDecimal
+import java.sql.{Date, Timestamp}
import java.time.{Duration, Period}
import scala.collection.JavaConverters._
+import org.apache.hadoop.fs.Path
+import org.apache.parquet.column.{Encoding, ParquetProperties}
import org.apache.parquet.hadoop.ParquetOutputFormat
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
+
Review comment:
removed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putByte(r, (byte) v));
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putShort(r, (short) v));
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putInt(r, (int) v));
+ }
+
+ // Based on VectorizedPlainValuesReader.readIntegersWithRebase
+ @Override
+ public final void readIntegersWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putInt(r, RebaseDateTime.rebaseJulianToGregorianDays((int) v));
+ }
+ } else {
+ w.putInt(r, (int) v);
+ }
+ });
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putLong(r, Integer.toUnsignedLong((int) v));
+ });
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putByteArray(r, new BigInteger(Long.toUnsignedString(v)).toByteArray());
+ });
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public final void readLongsWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putLong(r, RebaseDateTime.rebaseJulianToGregorianMicros(v));
+ }
+ } else {
+ w.putLong(r, v);
+ }
+ });
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ skipValues(total);
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ int remaining = total;
+ // First value
+ if (valuesRead == 0) {
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putByte(r, (byte) v));
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putShort(r, (short) v));
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putInt(r, (int) v));
+ }
+
+ // Based on VectorizedPlainValuesReader.readIntegersWithRebase
+ @Override
+ public final void readIntegersWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putInt(r, RebaseDateTime.rebaseJulianToGregorianDays((int) v));
+ }
+ } else {
+ w.putInt(r, (int) v);
+ }
+ });
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putLong(r, Integer.toUnsignedLong((int) v));
+ });
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putByteArray(r, new BigInteger(Long.toUnsignedString(v)).toByteArray());
+ });
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public final void readLongsWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putLong(r, RebaseDateTime.rebaseJulianToGregorianMicros(v));
+ }
+ } else {
+ w.putLong(r, v);
+ }
+ });
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ skipValues(total);
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putByte(r, (byte) v));
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putShort(r, (short) v));
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putInt(r, (int) v));
+ }
+
+ // Based on VectorizedPlainValuesReader.readIntegersWithRebase
+ @Override
+ public final void readIntegersWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putInt(r, RebaseDateTime.rebaseJulianToGregorianDays((int) v));
+ }
+ } else {
+ w.putInt(r, (int) v);
+ }
+ });
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putLong(r, Integer.toUnsignedLong((int) v));
+ });
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putByteArray(r, new BigInteger(Long.toUnsignedString(v)).toByteArray());
+ });
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public final void readLongsWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putLong(r, RebaseDateTime.rebaseJulianToGregorianMicros(v));
+ }
+ } else {
+ w.putLong(r, v);
+ }
+ });
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ skipValues(total);
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ int remaining = total;
+ // First value
+ if (valuesRead == 0) {
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putByte(r, (byte) v));
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putShort(r, (short) v));
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putInt(r, (int) v));
+ }
+
+ // Based on VectorizedPlainValuesReader.readIntegersWithRebase
+ @Override
+ public final void readIntegersWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putInt(r, RebaseDateTime.rebaseJulianToGregorianDays((int) v));
+ }
+ } else {
+ w.putInt(r, (int) v);
+ }
+ });
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putLong(r, Integer.toUnsignedLong((int) v));
+ });
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putByteArray(r, new BigInteger(Long.toUnsignedString(v)).toByteArray());
+ });
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public final void readLongsWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putLong(r, RebaseDateTime.rebaseJulianToGregorianMicros(v));
+ }
+ } else {
+ w.putLong(r, v);
+ }
+ });
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ skipValues(total);
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ int remaining = total;
+ // First value
+ if (valuesRead == 0) {
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putByte(r, (byte) v));
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putShort(r, (short) v));
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putInt(r, (int) v));
+ }
+
+ // Based on VectorizedPlainValuesReader.readIntegersWithRebase
+ @Override
+ public final void readIntegersWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putInt(r, RebaseDateTime.rebaseJulianToGregorianDays((int) v));
+ }
+ } else {
+ w.putInt(r, (int) v);
+ }
+ });
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putLong(r, Integer.toUnsignedLong((int) v));
+ });
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putByteArray(r, new BigInteger(Long.toUnsignedString(v)).toByteArray());
+ });
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public final void readLongsWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putLong(r, RebaseDateTime.rebaseJulianToGregorianMicros(v));
+ }
+ } else {
+ w.putLong(r, v);
+ }
+ });
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ skipValues(total);
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ int remaining = total;
+ // First value
+ if (valuesRead == 0) {
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 32 value each time
+ *
+ * see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ Arrays.fill(this.unpackedValuesBuffer, 0);
Review comment:
No we cannot. If there is a run of 0 deltas the packer used is Packer0 which does nothing i.e does not overwrite anything and the previous values will get used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994137755
**[Test build #146200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146200/testReport)** for PR 34471 at commit [`f472135`](https://github.com/apache/spark/commit/f472135a04a3129914b94ddddc03d506d07626a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999173130
If you click the SparkQA test build link you should see the failed tests. For instance:
```
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: "[no more values to read, total value count is 641]" did not equal "[No more values to read. Total values read: 641, total count: 641, trying to read 1 more.]"
at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
at org.apache.spark.sql.execution.datasources.parquet.ParquetDeltaEncodingSuite.$anonfun$new$21(ParquetDeltaEncodingSuite.scala:170)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r784271754
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -76,6 +79,7 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
saveAsCsvTable(testDf, dir.getCanonicalPath + "/csv")
saveAsJsonTable(testDf, dir.getCanonicalPath + "/json")
saveAsParquetTable(testDf, dir.getCanonicalPath + "/parquet")
+ saveAsParquetV2Table(testDf, dir.getCanonicalPath + "/parquetV2")
Review comment:
@LuciferYang I was getting ready to set up a PR for the RLE/Boolean encoding and noticed that you have done so. Thank you!
Adding back the benchmark in a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972677736
**[Test build #145367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145367/testReport)** for PR 34471 at commit [`2574e69`](https://github.com/apache/spark/commit/2574e69577285075f102d7a39ef373ced8bdaaea).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972578844
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49839/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972692219
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145367/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966816947
**[Test build #145122 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145122/testReport)** for PR 34471 at commit [`6021d90`](https://github.com/apache/spark/commit/6021d90a2a6bd9a90724bd30b4cffc999330f096).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase `
* `public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase `
* `public class VectorizedReaderBase extends ValuesReader implements VectorizedValuesReader `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966698908
**[Test build #145121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145121/testReport)** for PR 34471 at commit [`3572eaa`](https://github.com/apache/spark/commit/3572eaa42ce5360cc69b92888383bb39bef13a01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999764680
**[Test build #146491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146491/testReport)** for PR 34471 at commit [`64cc82f`](https://github.com/apache/spark/commit/64cc82fc2898cf553900b589f93bdd4a5887060d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r769047752
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -147,6 +159,16 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
}
}
+ sqlBenchmark.addCase("SQL Parquet Vectorized (Delta Binary)") { _ =>
Review comment:
Right. I have the RLE/boolean part implemented too. But need to add at least one unit test. I'll make that a different PR. Removing the benchmark for the moment.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingSuite.scala
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.{DeltaBinaryPackingValuesWriterForInteger, DeltaBinaryPackingValuesWriterForLong}
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{IntegerType, IntegralType, LongType}
+
+/**
+ * Read tests for vectorized Delta binary packed reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForLongTest
+ */
+abstract class ParquetDeltaEncodingSuite[T] extends ParquetCompatibilityTest
+ with SharedSparkSession {
+ protected var blockSize = 128
+ protected var miniBlockNum = 4
+ protected var reader: VectorizedDeltaBinaryPackedReader = _
+ protected var writableColumnVector: WritableColumnVector = _
+ protected var writer: ValuesWriter = _
+ protected var random: Random = _
+
+ protected def getSparkSqlType: IntegralType
+
+ protected def writeData(data: Array[T]): Unit
+
+ protected def writeData(data: Array[T], length: Int): Unit
+
+ protected def initValuesWriter (): Unit
+
+// protected def allocArrayOfType[T: ClassTag](size: Int): Array[T] = new Array[T](size)
Review comment:
yes. Done
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingSuite.scala
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.{DeltaBinaryPackingValuesWriterForInteger, DeltaBinaryPackingValuesWriterForLong}
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{IntegerType, IntegralType, LongType}
+
+/**
+ * Read tests for vectorized Delta binary packed reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForLongTest
+ */
+abstract class ParquetDeltaEncodingSuite[T] extends ParquetCompatibilityTest
+ with SharedSparkSession {
+ protected var blockSize = 128
+ protected var miniBlockNum = 4
+ protected var reader: VectorizedDeltaBinaryPackedReader = _
+ protected var writableColumnVector: WritableColumnVector = _
+ protected var writer: ValuesWriter = _
+ protected var random: Random = _
+
+ protected def getSparkSqlType: IntegralType
+
+ protected def writeData(data: Array[T]): Unit
+
+ protected def writeData(data: Array[T], length: Int): Unit
+
+ protected def initValuesWriter (): Unit
+
+// protected def allocArrayOfType[T: ClassTag](size: Int): Array[T] = new Array[T](size)
+
+ protected def allocDataArray(size: Int): Array[T]
+
+ protected def getNextRandom: T
+
+ protected def getTypeMinValue: T
+
+ protected def getTypeMaxValue: T
+
+ protected def readData(total: Int, columnVector : WritableColumnVector, rowId: Int): Unit
+
+ protected def skip(total: Int): Unit
+
+ protected def readDataFromVector(columnVector : WritableColumnVector, rowId: Int): T
+
+ protected def estimatedSize(length: Int) : Double
+
+ protected def setValue(arr: Array[T], index: Int, value: Int): Unit
+ protected def compareValues(expected: Int, actual: T) : Boolean
+
+
+ protected override def beforeEach(): Unit = {
+ blockSize = 128
+ miniBlockNum = 4
+ random = new Random(0)
+ initValuesWriter()
+ super.beforeAll()
+ }
+
+ test("read when data is aligned with block") {
+ val data = allocDataArray(5 * blockSize)
+ for (i <- 0 until blockSize * 5) {
+ data(i) = getNextRandom
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when block is not fully written") {
+ val data = allocDataArray(blockSize - 3)
+ for (i <- data.indices) {
+ data(i) = getNextRandom
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when mini block is not fully written") {
+ val miniBlockSize = blockSize / miniBlockNum
+ val data = allocDataArray(miniBlockSize - 3)
+ for (i <- data.indices) {
+ data(i) = getNextRandom
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read with negative deltas") {
+ val data = allocDataArray(blockSize)
+ for (i <- data.indices) {
+ setValue(data, i, 10 - (i * 32 - random.nextInt(6)))
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when deltas are same") {
+ val data = allocDataArray(2 * blockSize)
+ for (i <- 0 until blockSize) {
+ setValue(data, i, i * 32)
+ }
+ for (i <- blockSize until 2 * blockSize) {
+ setValue(data, i, 0)
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when values are same") {
+ val data = allocDataArray(2 * blockSize)
+ for (i <- 0 until blockSize) {
+ setValue(data, i, 3)
+ }
+ for (i <- blockSize until 2 * blockSize) {
+ setValue(data, i, 0)
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when delta is 0 for each block") {
+ val data = allocDataArray(5 * blockSize + 1)
+ for (i <- data.indices) {
+ setValue(data, i, (i - 1) / blockSize)
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when data is not aligned with block") {
+ val data = allocDataArray(5 * blockSize + 3)
+ for (i <- data.indices) {
+ setValue(data, i, random.nextInt(20) - 10)
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read max min value") {
+ val data = allocDataArray(10)
+ for (i <- data.indices) {
+ if (i % 2 == 0) data(i) = getTypeMinValue
+ else data(i) = getTypeMaxValue
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("throw exception when read more than written") {
+ val data = allocDataArray(5 * blockSize + 1)
+ for (i <- data.indices) {
+ setValue(data, i, i * 32)
+ }
+ shouldWriteAndRead(data)
+ try readData(1, writableColumnVector, data.length)
+ catch {
+ case e: ParquetDecodingException =>
+ assert("no more values to read, total value count is " + data.length == e.getMessage)
+ }
+ }
+
+ test("skip()") {
+ val data = allocDataArray(5 * blockSize + 1)
+ for (i <- data.indices) {
+ setValue(data, i, i * 32)
+ }
+ writeData(data)
+ reader = new VectorizedDeltaBinaryPackedReader
+ reader.initFromPage(100, writer.getBytes.toInputStream)
+ writableColumnVector = new OnHeapColumnVector(data.length, getSparkSqlType)
+ for (i <- data.indices) {
+ if (i % 3 == 0) {
+ skip(1)
+ } else {
+ readData(1, writableColumnVector, i)
+ assert(compareValues(i * 32, readDataFromVector(writableColumnVector, i)))
+ }
+ }
+ }
+
+ test("SkipN()") {
+ val data = allocDataArray(5 * blockSize + 1)
+ for (i <- data.indices) {
+ setValue(data, i, i * 32)
+ }
+ writeData(data)
+ reader = new VectorizedDeltaBinaryPackedReader
+ reader.initFromPage(100, writer.getBytes.toInputStream)
+ writableColumnVector = new OnHeapColumnVector(data.length, getSparkSqlType)
+ var skipCount = 0
+ var i = 0
+ while (i < data.length) {
+ skipCount = (data.length - i) / 2
+ readData(1, writableColumnVector, i)
+ assert(compareValues(i * 32, readDataFromVector(writableColumnVector, i)))
+ skip(skipCount)
+
+ i += skipCount + 1
+ }
+ }
+
+ test("random data test") {
+ val maxSize = 1000
+ val data = allocDataArray(maxSize)
+ for (round <- 0 until 100000) {
+ val size = random.nextInt(maxSize)
+ for (i <- 0 until size) {
+ data(i) = getNextRandom
+ }
+ shouldReadAndWrite(data, size)
+ writer.reset()
+ }
+ }
+
+ @throws[IOException]
+ private def shouldWriteAndRead(data: Array[T]): Unit = {
+ shouldReadAndWrite(data, data.length)
+ }
+
+ private def shouldReadAndWrite(data: Array[T], length: Int): Unit = {
+ writeData(data, length)
+ reader = new VectorizedDeltaBinaryPackedReader
+ val page = writer.getBytes.toByteArray
+
+ assert(estimatedSize(length) >= page.length)
+ writableColumnVector = new OnHeapColumnVector(data.length, getSparkSqlType)
+ reader.initFromPage(100, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)))
+ readData(length, writableColumnVector, 0)
+ for (i <- 0 until length) {
+ assert(data(i) == readDataFromVector(writableColumnVector, i))
+ }
+ }
+
+}
+
+class ParquetDeltaEncodingInteger extends ParquetDeltaEncodingSuite[Int] {
+
+ override protected def getSparkSqlType: IntegralType = IntegerType
+ override protected def writeData(data: Array[Int]): Unit = writeData(data, data.length)
+
+ override protected def writeData(data: Array[Int], length: Int): Unit = {
+ for (i <- 0 until length) {
+ writer.writeInteger(data(i))
+ }
+ }
+
+ override protected def initValuesWriter (): Unit = {
+ writer = new DeltaBinaryPackingValuesWriterForInteger(
+ blockSize,
+ miniBlockNum,
+ 100,
+ 200,
+ new DirectByteBufferAllocator())
+ }
+
+ override protected def allocDataArray(size: Int): Array[Int] = new Array[Int](size)
+
+ override protected def getNextRandom: Int = random.nextInt
+ override protected def getTypeMinValue: Int = Int.MinValue
+ override protected def getTypeMaxValue: Int = Int.MaxValue
+
+ override protected def readData(total: Int, columnVector : WritableColumnVector, rowId: Int): Unit
+ = reader.readIntegers(total, columnVector, rowId)
+
+ override protected def skip(total: Int): Unit = reader.skipIntegers(total)
+
+ override protected def readDataFromVector(columnVector: WritableColumnVector, rowId: Int): Int =
+ columnVector.getInt(rowId)
+
+ override protected def estimatedSize(length: Int) : Double = {
+ val miniBlockSize = blockSize / miniBlockNum
+ val miniBlockFlushed = Math.ceil((length.toDouble - 1) / miniBlockSize)
+ val blockFlushed = Math.ceil((length.toDouble - 1) / blockSize)
+ 4 * 5 /* blockHeader */ +
+ 4 * miniBlockFlushed * miniBlockSize /* data(aligned to miniBlock) */ +
+ blockFlushed * miniBlockNum /* bitWidth of mini blocks */ +
+ (5.0 * blockFlushed) /* min delta for each block */
+ }
+
+ override protected def setValue(arr: Array[Int], index: Int, value: Int): Unit =
+ arr(index) = value
+
+ override protected def compareValues(expected: Int, actual: Int) : Boolean =
+ expected == actual
+
+}
+
+class ParquetDeltaEncodingLong extends ParquetDeltaEncodingSuite[Long] {
+
+ override protected def getSparkSqlType: IntegralType = LongType
+ override protected def writeData(data: Array[Long]): Unit = writeData(data, data.length)
+
+ override protected def writeData(data: Array[Long], length: Int): Unit = {
+ for (i <- 0 until length) {
+ writer.writeLong(data(i))
+ }
+ }
+
+ override protected def initValuesWriter (): Unit = {
+ writer = new DeltaBinaryPackingValuesWriterForLong(
+ blockSize,
+ miniBlockNum,
+ 100,
+ 200,
+ new DirectByteBufferAllocator())
+ }
+
+ override protected def allocDataArray(size: Int): Array[Long] = new Array[Long](size)
+
+ override protected def getNextRandom: Long = random.nextLong
+ override protected def getTypeMinValue: Long = Long.MinValue
+ override protected def getTypeMaxValue: Long = Long.MaxValue
+
+ override protected def readData(total: Int, columnVector: WritableColumnVector, rowId: Int): Unit
+ = reader.readLongs(total, columnVector, rowId)
Review comment:
OK.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingSuite.scala
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.{DeltaBinaryPackingValuesWriterForInteger, DeltaBinaryPackingValuesWriterForLong}
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{IntegerType, IntegralType, LongType}
+
+/**
+ * Read tests for vectorized Delta binary packed reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForLongTest
+ */
+abstract class ParquetDeltaEncodingSuite[T] extends ParquetCompatibilityTest
+ with SharedSparkSession {
+ protected var blockSize = 128
+ protected var miniBlockNum = 4
+ protected var reader: VectorizedDeltaBinaryPackedReader = _
+ protected var writableColumnVector: WritableColumnVector = _
+ protected var writer: ValuesWriter = _
+ protected var random: Random = _
+
+ protected def getSparkSqlType: IntegralType
+
+ protected def writeData(data: Array[T]): Unit
+
+ protected def writeData(data: Array[T], length: Int): Unit
+
+ protected def initValuesWriter (): Unit
+
+// protected def allocArrayOfType[T: ClassTag](size: Int): Array[T] = new Array[T](size)
+
+ protected def allocDataArray(size: Int): Array[T]
+
+ protected def getNextRandom: T
+
+ protected def getTypeMinValue: T
+
+ protected def getTypeMaxValue: T
+
+ protected def readData(total: Int, columnVector : WritableColumnVector, rowId: Int): Unit
+
+ protected def skip(total: Int): Unit
+
+ protected def readDataFromVector(columnVector : WritableColumnVector, rowId: Int): T
+
+ protected def estimatedSize(length: Int) : Double
+
+ protected def setValue(arr: Array[T], index: Int, value: Int): Unit
+ protected def compareValues(expected: Int, actual: T) : Boolean
+
+
+ protected override def beforeEach(): Unit = {
+ blockSize = 128
Review comment:
Removed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column.
+ * <p>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2.
+ * <p>
+ * Supported Types: INT32, INT64
+ * <p>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ // currentMiniBlock keeps track of the mini block within the current block that
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putByte(r, (byte) v));
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putShort(r, (short) v));
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putInt(r, (int) v));
+ }
+
+ // Based on VectorizedPlainValuesReader.readIntegersWithRebase
+ @Override
+ public final void readIntegersWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putInt(r, RebaseDateTime.rebaseJulianToGregorianDays((int) v));
+ }
+ } else {
+ w.putInt(r, (int) v);
+ }
+ });
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putLong(r, Integer.toUnsignedLong((int) v));
+ });
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putByteArray(r, new BigInteger(Long.toUnsignedString(v)).toByteArray());
+ });
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public final void readLongsWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianTs()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putLong(r, RebaseDateTime.rebaseJulianToGregorianMicros(v));
+ }
+ } else {
+ w.putLong(r, v);
+ }
+ });
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ skipValues(total);
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "No more values to read, total value count is " + valuesRead);
Review comment:
done
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -142,13 +154,23 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
}
}
+ sqlBenchmark.addCase("SQL Parquet Vectorized (Delta Binary)") { _ =>
+ spark.sql("select sum(id) from parquetV2Table").noop()
+ }
+
+ sqlBenchmark.addCase("SQL Parquet MR (Delta Binary)") { _ =>
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ spark.sql("select sum(id) from parquetV2Table").noop()
+ }
+ }
+
sqlBenchmark.addCase("SQL ORC Vectorized") { _ =>
- spark.sql(s"SELECT $query FROM orcTable").noop()
+ spark.sql("SELECT sum(id) FROM orcTable").noop()
Review comment:
Uh. Rebase issue. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959699641
Jenkins, retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958554880
**[Test build #144861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144861/testReport)** for PR 34471 at commit [`fc9683b`](https://github.com/apache/spark/commit/fc9683bfc0af15050ddfdeef50a9ff7a45078bb8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959709152
**[Test build #144880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144880/testReport)** for PR 34471 at commit [`10659d3`](https://github.com/apache/spark/commit/10659d3bc9c1f3b3b9f8c29000f9f75e95281777).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959814220
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49350/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958531680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-963633817
Looks like it may take some time to address some of the review comments. Marking this PR as draft in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-992849326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-992810207
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50617/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-992843741
**[Test build #146144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146144/testReport)** for PR 34471 at commit [`91f7534`](https://github.com/apache/spark/commit/91f7534c87c67e6e456a06f6f250dd63e8b14f3c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994203174
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146200/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966784677
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49596/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972526301
> Looks much better now @parthchandra ! do you have any benchmark result to share, i.e., comparing the vectorized reader versus non-vectorized reader from `parquet-mr`? you can use `DataSourceReadBenchmark` for this.
I've added a couple of cases to the benchmark for this. Because Delta encoding is more suited for sorted data, I also ran a modified version (not checked in) on sorted data.
TL;DR For unsorted data Delta vectorized is slower than default (RLE) vectorized reader. For sorted data, Delta vectorized is much faster than unsorted data, but is still slightly slower than default (RLE) vectorized reader. For both unsorted/sorted data the vectorized reader is ~7x/~10x faster than the Parquet-MR reader.
Here is the summary for INT data
UNSORTED
```
OpenJDK 64-Bit Server VM 11.0.6+10-LTS on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
SQL CSV 12625 12720 135 1.2 802.7 1.0X
SQL Json 6784 6844 85 2.3 431.3 1.9X
SQL Parquet Vectorized 106 112 7 148.5 6.7 119.2X
SQL Parquet MR 1594 1601 9 9.9 101.4 7.9X
SQL ORC Vectorized 218 227 5 72.0 13.9 57.8X
SQL ORC MR 1289 1320 44 12.2 81.9 9.8X
SQL Parquet Vectorized (Delta Binary) 232 243 17 67.9 14.7 54.5X
SQL Parquet MR (Delta Binary) 1462 1478 24 10.8 92.9 8.6X
```
SORTED
```
OpenJDK 64-Bit Server VM 11.0.6+10-LTS on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
SQL CSV 12135 12150 22 1.3 771.5 1.0X
SQL Json 6138 6177 54 2.6 390.3 2.0X
SQL Parquet Vectorized 102 107 7 153.5 6.5 118.4X
SQL Parquet MR 1489 1496 9 10.6 94.7 8.1X
SQL ORC Vectorized 166 171 3 94.9 10.5 73.2X
SQL ORC MR 1184 1188 5 13.3 75.3 10.3X
SQL Parquet Vectorized (Delta Binary) 130 138 8 120.7 8.3 93.1X
SQL Parquet MR (Delta Binary) 1287 1306 26 12.2 81.8 9.4X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958531680
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49330/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r743047058
##########
File path: .gitignore
##########
@@ -33,6 +33,7 @@ R/pkg/tests/fulltests/Rplots.pdf
build/*.jar
build/apache-maven*
build/scala*
+build/zinc*
Review comment:
nit: unrelated change?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
+ }
+
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
Review comment:
this doesn't seem to be used anywhere
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
Review comment:
maybe add some comments to these fields - it's a bit confusing that `currentMiniBlock` is an integer
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ for (int i = 0; i < total; i++) {
+ Binary binary = deltaByteArrayReader.readBytes();
+ ByteBuffer buffer = binary.toByteBuffer();
+ if (buffer.hasArray()) {
+ c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
+ binary.length());
+ } else {
+ byte[] bytes = new byte[binary.length()];
+ buffer.get(bytes);
+ c.putByteArray(rowId + i, bytes);
+ }
+ }
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+
+ }
+
+ @Override
+ public void skipBytes(int total) {
+
+ }
+
+ @Override
+ public void skipShorts(int total) {
+
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+
+ }
+
+ @Override
+ public void skipLongs(int total) {
+
+ }
+
+ @Override
+ public void skipFloats(int total) {
+
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+
+ }
+
+ @Override
+ public void skipBinary(int total) {
Review comment:
we need this too
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -166,7 +166,7 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
readState.resetForNewPage(pageValueCount, pageFirstRowIndex);
}
PrimitiveType.PrimitiveTypeName typeName =
- descriptor.getPrimitiveType().getPrimitiveTypeName();
+ descriptor.getPrimitiveType().getPrimitiveTypeName();
Review comment:
ditto
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
Review comment:
seems it's better to have an abstract class inheriting `ValuesReader` and `VectorizedValuesReader` with this default behavior defined, rather than repeating the same thing in all the different value readers.
this can be done separately though.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
Review comment:
hmm I wonder why we decrement `remaining` here.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
Review comment:
should this be `w.putInt`?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
Review comment:
maybe add some comments for this? what are `c`, `rowId` and `val` for?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+
Review comment:
ditto
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
Review comment:
nit: add space after `//`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ BytePackerForLong packer = Packer.LITTLE_ENDIAN.newBytePackerForLong(
+ bitWidths[currentMiniBlock]);
+ for (int j = 0; j < miniBlockSizeInValues; j += 8) {
+ ByteBuffer buffer = in.slice(packer.getBitWidth());
+ packer.unpack8Values(buffer, buffer.position(), unpackedValuesBuffer, j);
Review comment:
wonder if we can unpack 32 values at once, since the [spec](https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5) says the number of values in a mini block is a multiple of 32.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
Review comment:
nit: remove `/*unused*/`: it's used although just in precondition check
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
Review comment:
maybe add a bit more information, like when it will be used, what data types it supports, etc
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
Review comment:
maybe we can reuse `this.unpackedValuesBuffer`?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -122,4 +126,40 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
}
}
}
+
+ test("parquet v2 pages - delta encoding") {
+ val extraOptions = Map[String, String](
+ ParquetOutputFormat.WRITER_VERSION -> ParquetProperties.WriterVersion.PARQUET_2_0.toString,
+ ParquetOutputFormat.ENABLE_DICTIONARY -> "false"
+ )
+
+ val hadoopConf = spark.sessionState.newHadoopConfWithOptions(extraOptions)
+ withSQLConf(
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",
+ ParquetOutputFormat.JOB_SUMMARY_LEVEL -> "ALL") {
Review comment:
why we need this?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingIntegerSuite.scala
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForInteger
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.IntegerType
+
+/**
+ * Read tests for vectorized Delta binary packed Integer reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ */
+class ParquetDeltaEncodingIntegerSuite extends ParquetCompatibilityTest with SharedSparkSession {
Review comment:
can we combine this with `ParquetDeltaEncodingLongSuite`? we can have a base class with the common code (need to abstract a few things like how to allocate an array, how to generate a rand int/long, and Parquet's value writer class etc.)
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
Review comment:
why return 0 here?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
Review comment:
I think we'll need to implement these too.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
Review comment:
I think this can be done more efficiently, for instance we don't need to unpack the bits anymore, and don't need to compute the original value from delta, etc.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
Review comment:
nit: remove
##########
File path: .gitignore
##########
@@ -33,6 +33,7 @@ R/pkg/tests/fulltests/Rplots.pdf
build/*.jar
build/apache-maven*
build/scala*
+build/zinc*
Review comment:
nit: unrelated change?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
+ }
+
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
Review comment:
this doesn't seem to be used anywhere
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
Review comment:
maybe add some comments to these fields - it's a bit confusing that `currentMiniBlock` is an integer
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ for (int i = 0; i < total; i++) {
+ Binary binary = deltaByteArrayReader.readBytes();
+ ByteBuffer buffer = binary.toByteBuffer();
+ if (buffer.hasArray()) {
+ c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
+ binary.length());
+ } else {
+ byte[] bytes = new byte[binary.length()];
+ buffer.get(bytes);
+ c.putByteArray(rowId + i, bytes);
+ }
+ }
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+
+ }
+
+ @Override
+ public void skipBytes(int total) {
+
+ }
+
+ @Override
+ public void skipShorts(int total) {
+
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+
+ }
+
+ @Override
+ public void skipLongs(int total) {
+
+ }
+
+ @Override
+ public void skipFloats(int total) {
+
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+
+ }
+
+ @Override
+ public void skipBinary(int total) {
Review comment:
we need this too
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -166,7 +166,7 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
readState.resetForNewPage(pageValueCount, pageFirstRowIndex);
}
PrimitiveType.PrimitiveTypeName typeName =
- descriptor.getPrimitiveType().getPrimitiveTypeName();
+ descriptor.getPrimitiveType().getPrimitiveTypeName();
Review comment:
ditto
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
Review comment:
seems it's better to have an abstract class inheriting `ValuesReader` and `VectorizedValuesReader` with this default behavior defined, rather than repeating the same thing in all the different value readers.
this can be done separately though.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
Review comment:
hmm I wonder why we decrement `remaining` here.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
Review comment:
should this be `w.putInt`?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
Review comment:
maybe add some comments for this? what are `c`, `rowId` and `val` for?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+
Review comment:
ditto
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
Review comment:
nit: add space after `//`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ BytePackerForLong packer = Packer.LITTLE_ENDIAN.newBytePackerForLong(
+ bitWidths[currentMiniBlock]);
+ for (int j = 0; j < miniBlockSizeInValues; j += 8) {
+ ByteBuffer buffer = in.slice(packer.getBitWidth());
+ packer.unpack8Values(buffer, buffer.position(), unpackedValuesBuffer, j);
Review comment:
wonder if we can unpack 32 values at once, since the [spec](https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5) says the number of values in a mini block is a multiple of 32.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
Review comment:
nit: remove `/*unused*/`: it's used although just in precondition check
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
Review comment:
maybe add a bit more information, like when it will be used, what data types it supports, etc
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
Review comment:
maybe we can reuse `this.unpackedValuesBuffer`?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -122,4 +126,40 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
}
}
}
+
+ test("parquet v2 pages - delta encoding") {
+ val extraOptions = Map[String, String](
+ ParquetOutputFormat.WRITER_VERSION -> ParquetProperties.WriterVersion.PARQUET_2_0.toString,
+ ParquetOutputFormat.ENABLE_DICTIONARY -> "false"
+ )
+
+ val hadoopConf = spark.sessionState.newHadoopConfWithOptions(extraOptions)
+ withSQLConf(
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",
+ ParquetOutputFormat.JOB_SUMMARY_LEVEL -> "ALL") {
Review comment:
why we need this?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingIntegerSuite.scala
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForInteger
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.IntegerType
+
+/**
+ * Read tests for vectorized Delta binary packed Integer reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ */
+class ParquetDeltaEncodingIntegerSuite extends ParquetCompatibilityTest with SharedSparkSession {
Review comment:
can we combine this with `ParquetDeltaEncodingLongSuite`? we can have a base class with the common code (need to abstract a few things like how to allocate an array, how to generate a rand int/long, and Parquet's value writer class etc.)
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
Review comment:
why return 0 here?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
Review comment:
I think we'll need to implement these too.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
Review comment:
I think this can be done more efficiently, for instance we don't need to unpack the bits anymore, and don't need to compute the original value from delta, etc.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
Review comment:
nit: remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r779409193
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column.
+ * <p>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2.
+ * <p>
+ * Supported Types: INT32, INT64
+ * <p>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ // currentMiniBlock keeps track of the mini block within the current block that
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
Review comment:
Should these 4 field be private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r779406568
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -76,6 +79,7 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
saveAsCsvTable(testDf, dir.getCanonicalPath + "/csv")
saveAsJsonTable(testDf, dir.getCanonicalPath + "/json")
saveAsParquetTable(testDf, dir.getCanonicalPath + "/parquet")
+ saveAsParquetV2Table(testDf, dir.getCanonicalPath + "/parquetV2")
Review comment:
~~Maybe we should update the benchmark-result of `DataSourceReadBenchmark`~~ @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-961492262
cc @sadikovi @viirya @dongjoon-hyun too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r751550718
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column. <br/>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2. <br/>
+ * Supported Types: INT32, INT64 <br/>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ private int currentMiniBlock = 0; // keep track of the mini block within the current block that we
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @SuppressWarnings("unused")
Review comment:
Removed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
Review comment:
Sorry, my previous comment was for a different part of the code.
You're right, remaining does not need to be decremented. Doing so was just shortening the loop so loadMiniBlockToOutput was being called more times than necessary.
Nice catch. Thank you!
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
##########
@@ -168,6 +169,16 @@ private[sql] trait ParquetTest extends FileBasedDataSourceTest {
withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true")(code)
}
+ def withAllParquetWriters(code: => Unit): Unit = {
+ ParquetOutputFormat.WRITER_VERSION -> ParquetProperties.WriterVersion.PARQUET_2_0.toString
Review comment:
Oops. Cut-paste error. Removed.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
+ }
+
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
Review comment:
I just knew you would say that :). Done.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedReaderBase.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * Base class for implementations of VectorizedValuesReader. Mainly to avoid duplication
+ * of methods that are not supported by concrete implementations
+ */
+public class VectorizedReaderBase extends ValuesReader implements VectorizedValuesReader {
Review comment:
Yeah, I thought about that too. But then I left it so as not to clutter up the PR.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ return deltaByteArrayReader.readBytes();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ for (int i = 0; i < total; i++) {
+ Binary binary = deltaByteArrayReader.readBytes();
+ ByteBuffer buffer = binary.toByteBuffer();
+ if (buffer.hasArray()) {
+ c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
+ binary.length());
+ } else {
+ byte[] bytes = new byte[binary.length()];
+ buffer.get(bytes);
+ c.putByteArray(rowId + i, bytes);
+ }
+ }
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ for(int i =0; i < total; i++) {
Review comment:
Fixed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
Review comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958302148
**[Test build #144861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144861/testReport)** for PR 34471 at commit [`fc9683b`](https://github.com/apache/spark/commit/fc9683bfc0af15050ddfdeef50a9ff7a45078bb8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958557735
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144861/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958547677
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49334/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958606731
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144864/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958302148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958531680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r744037817
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
Review comment:
The import order is a little strange. Could you grouping `java` import (line 30 and 19) together as the first group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966761997
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966807372
**[Test build #145125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145125/testReport)** for PR 34471 at commit [`a88c721`](https://github.com/apache/spark/commit/a88c72169e7c3e624db1303ab5db1bfac253eb37).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase `
* `public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase `
* `public class VectorizedReaderBase extends ValuesReader implements VectorizedValuesReader `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994203174
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146200/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-992849326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r779825155
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRebaseDatetimeSuite.scala
##########
@@ -134,33 +134,35 @@ abstract class ParquetRebaseDatetimeSuite
tsOutputType: String = "TIMESTAMP_MICROS",
inWriteConf: String = SQLConf.PARQUET_REBASE_MODE_IN_WRITE.key,
inReadConf: String = SQLConf.PARQUET_REBASE_MODE_IN_READ.key): Unit = {
- withTempPaths(2) { paths =>
- paths.foreach(_.delete())
+ withAllParquetWriters {
+ withTempPaths(2) { paths =>
+ paths.foreach(_.delete())
val oldPath = getResourceParquetFilePath("test-data/" + fileName)
val path3_x = paths(0).getCanonicalPath
val path3_x_rebase = paths(1).getCanonicalPath
- val df = Seq.tabulate(N)(rowFunc).toDF("dict", "plain")
- .select($"dict".cast(catalystType), $"plain".cast(catalystType))
- withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> tsOutputType) {
+ val df = Seq.tabulate(N)(rowFunc).toDF("dict", "plain")
+ .select($"dict".cast(catalystType), $"plain".cast(catalystType))
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> tsOutputType) {
Review comment:
Something wrong with indentation here and below:
<img width="642" alt="Screenshot 2022-01-06 at 22 56 02" src="https://user-images.githubusercontent.com/1580697/148443305-f6890111-26c2-4538-9abe-53a28144c46b.png">
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRebaseDatetimeSuite.scala
##########
@@ -219,60 +221,62 @@ abstract class ParquetRebaseDatetimeSuite
test("SPARK-31159, SPARK-37705: rebasing timestamps in write") {
val N = 8
Seq(false, true).foreach { dictionaryEncoding =>
- Seq(
- (
- "TIMESTAMP_MILLIS",
- "1001-01-01 01:02:03.123",
- "1001-01-07 01:09:05.123",
- SQLConf.PARQUET_REBASE_MODE_IN_WRITE.key,
- SQLConf.PARQUET_REBASE_MODE_IN_READ.key),
- (
- "TIMESTAMP_MICROS",
- "1001-01-01 01:02:03.123456",
- "1001-01-07 01:09:05.123456",
- SQLConf.PARQUET_REBASE_MODE_IN_WRITE.key,
- SQLConf.PARQUET_REBASE_MODE_IN_READ.key),
- (
- "INT96",
- "1001-01-01 01:02:03.123456",
- "1001-01-07 01:09:05.123456",
- SQLConf.PARQUET_INT96_REBASE_MODE_IN_WRITE.key,
- SQLConf.PARQUET_INT96_REBASE_MODE_IN_READ.key
- )
- ).foreach { case (outType, tsStr, nonRebased, inWriteConf, inReadConf) =>
+ withAllParquetWriters {
+ Seq(
+ (
+ "TIMESTAMP_MILLIS",
+ "1001-01-01 01:02:03.123",
+ "1001-01-07 01:09:05.123",
+ SQLConf.PARQUET_REBASE_MODE_IN_WRITE.key,
+ SQLConf.PARQUET_REBASE_MODE_IN_READ.key),
+ (
+ "TIMESTAMP_MICROS",
+ "1001-01-01 01:02:03.123456",
+ "1001-01-07 01:09:05.123456",
+ SQLConf.PARQUET_REBASE_MODE_IN_WRITE.key,
+ SQLConf.PARQUET_REBASE_MODE_IN_READ.key),
+ (
+ "INT96",
+ "1001-01-01 01:02:03.123456",
+ "1001-01-07 01:09:05.123456",
+ SQLConf.PARQUET_INT96_REBASE_MODE_IN_WRITE.key,
+ SQLConf.PARQUET_INT96_REBASE_MODE_IN_READ.key
+ )
+ ).foreach { case (outType, tsStr, nonRebased, inWriteConf, inReadConf) =>
// Ignore the default JVM time zone and use the session time zone instead of it in rebasing.
DateTimeTestUtils.withDefaultTimeZone(DateTimeTestUtils.JST) {
withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> DateTimeTestUtils.LA.getId) {
- withClue(s"output type $outType") {
- withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> outType) {
- withTempPath { dir =>
- val path = dir.getAbsolutePath
- withSQLConf(inWriteConf -> LEGACY.toString) {
- Seq.tabulate(N)(_ => tsStr).toDF("tsS")
- .select($"tsS".cast("timestamp").as("ts"))
- .repartition(1)
- .write
- .option("parquet.enable.dictionary", dictionaryEncoding)
- .parquet(path)
- }
+ withClue(s"output type $outType") {
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> outType) {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ withSQLConf(inWriteConf -> LEGACY.toString) {
+ Seq.tabulate(N)(_ => tsStr).toDF("tsS")
+ .select($"tsS".cast("timestamp").as("ts"))
+ .repartition(1)
+ .write
+ .option("parquet.enable.dictionary", dictionaryEncoding)
+ .parquet(path)
+ }
- withAllParquetReaders {
+ withAllParquetReaders {
// The file metadata indicates if it needs rebase or not, so we can always get
// the correct result regardless of the "rebase mode" config.
- runInMode(inReadConf, Seq(LEGACY, CORRECTED, EXCEPTION)) { options =>
- checkAnswer(
+ runInMode(inReadConf, Seq(LEGACY, CORRECTED, EXCEPTION)) { options =>
+ checkAnswer(
spark.read.options(options).parquet(path).select($"ts".cast("string")),
Seq.tabulate(N)(_ => Row(tsStr)))
- }
+ }
- // Force to not rebase to prove the written datetime values are rebased
- // and we will get wrong result if we don't rebase while reading.
- withSQLConf("spark.test.forceNoRebase" -> "true") {
- checkAnswer(
+ // Force to not rebase to prove the written datetime values are rebased
+ // and we will get wrong result if we don't rebase while reading.
+ withSQLConf("spark.test.forceNoRebase" -> "true") {
+ checkAnswer(
spark.read.parquet(path).select($"ts".cast("string")),
Seq.tabulate(N)(_ => Row(nonRebased)))
}
}
+ }
Review comment:
Wrong indentation too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966732464
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49592/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972575765
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49839/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972531356
**[Test build #145367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145367/testReport)** for PR 34471 at commit [`2574e69`](https://github.com/apache/spark/commit/2574e69577285075f102d7a39ef373ced8bdaaea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994137755
**[Test build #146200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146200/testReport)** for PR 34471 at commit [`f472135`](https://github.com/apache/spark/commit/f472135a04a3129914b94ddddc03d506d07626a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-992741489
**[Test build #146144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146144/testReport)** for PR 34471 at commit [`91f7534`](https://github.com/apache/spark/commit/91f7534c87c67e6e456a06f6f250dd63e8b14f3c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-992774603
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50617/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958531680
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49330/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999791362
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50967/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999945283
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146491/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r780024037
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -76,6 +79,7 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
saveAsCsvTable(testDf, dir.getCanonicalPath + "/csv")
saveAsJsonTable(testDf, dir.getCanonicalPath + "/json")
saveAsParquetTable(testDf, dir.getCanonicalPath + "/parquet")
+ saveAsParquetV2Table(testDf, dir.getCanonicalPath + "/parquetV2")
Review comment:
I found that there are still unsupported encoding in Data Page V2, such as RLE. It seems that it is not time to update the benchmark, please ignore my previous comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-961492262
cc @sadikovi @viirya @dongjoon-hyun too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r744038286
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
Review comment:
nit. Let's remove redundant empty line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959871361
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144880/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958446671
**[Test build #144864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144864/testReport)** for PR 34471 at commit [`10659d3`](https://github.com/apache/spark/commit/10659d3bc9c1f3b3b9f8c29000f9f75e95281777).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959757428
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49350/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958589512
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49334/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958589512
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49334/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994160993
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50674/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-1005946804
Committed to master branch, thanks @parthchandra !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao closed pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao closed pull request #34471:
URL: https://github.com/apache/spark/pull/34471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r744973480
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
Review comment:
done
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -122,4 +126,40 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
}
}
}
+
+ test("parquet v2 pages - delta encoding") {
+ val extraOptions = Map[String, String](
+ ParquetOutputFormat.WRITER_VERSION -> ParquetProperties.WriterVersion.PARQUET_2_0.toString,
+ ParquetOutputFormat.ENABLE_DICTIONARY -> "false"
+ )
+
+ val hadoopConf = spark.sessionState.newHadoopConfWithOptions(extraOptions)
+ withSQLConf(
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",
+ ParquetOutputFormat.JOB_SUMMARY_LEVEL -> "ALL") {
Review comment:
It does seem unnecessary, but I removed it and the test broke. Turns out this writes a metadata summary file that is read by `readFooter`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -280,6 +277,20 @@ private void initDataReader(
}
}
+ private ValuesReader getValuesReader(Encoding encoding) {
+ switch (encoding) {
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ BytePackerForLong packer = Packer.LITTLE_ENDIAN.newBytePackerForLong(
+ bitWidths[currentMiniBlock]);
+ for (int j = 0; j < miniBlockSizeInValues; j += 8) {
+ ByteBuffer buffer = in.slice(packer.getBitWidth());
+ packer.unpack8Values(buffer, buffer.position(), unpackedValuesBuffer, j);
Review comment:
I had thought about that too. I had looked at `ByteBitPackingForLongLE.unpack8Values` and my head hurt, so I left it. But on a second look, it does appear to be a simple change. I made the change and everything worked (i.e. all the unit tests pass)
However, looking at the implementation of the DeltaBinaryPackedWriter it turns out that the number of values in a mini block is assumed to be a multiple of 8 (not 32). (Also see: https://issues.apache.org/jira/browse/PARQUET-2077)
So it is probably safer to leave it as it is.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
Review comment:
I guess we could. Essentially, we need an initialized array of values and I was lazy and just allocated a new one.
TL;DR: In case of a run of deltas that are zero, the bitwidth for the miniblock is zero and the `Packer` used to decode is a `Packer0` which basically does nothing, leaving the output buffer untouched. If we reuse the buffer without re-initializing it, the old incorrect value gets used.
Replaced with array initialized to zero on each use.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
Review comment:
Done
##########
File path: .gitignore
##########
@@ -33,6 +33,7 @@ R/pkg/tests/fulltests/Rplots.pdf
build/*.jar
build/apache-maven*
build/scala*
+build/zinc*
Review comment:
Removed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
Review comment:
`remaining` is the number values left to be written. We've just written the first value here so `remaining` is now `total - 1`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -166,7 +166,7 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
readState.resetForNewPage(pageValueCount, pageFirstRowIndex);
}
PrimitiveType.PrimitiveTypeName typeName =
- descriptor.getPrimitiveType().getPrimitiveTypeName();
+ descriptor.getPrimitiveType().getPrimitiveTypeName();
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
Review comment:
yes. Good catch. Corrected.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
Review comment:
Fixed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
Review comment:
Why indeed. (Intellij generated code for unimplemented methods)
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ for (int i = 0; i < total; i++) {
+ Binary binary = deltaByteArrayReader.readBytes();
+ ByteBuffer buffer = binary.toByteBuffer();
+ if (buffer.hasArray()) {
+ c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
+ binary.length());
+ } else {
+ byte[] bytes = new byte[binary.length()];
+ buffer.get(bytes);
+ c.putByteArray(rowId + i, bytes);
+ }
+ }
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+
+ }
+
+ @Override
+ public void skipBytes(int total) {
+
+ }
+
+ @Override
+ public void skipShorts(int total) {
+
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+
+ }
+
+ @Override
+ public void skipLongs(int total) {
+
+ }
+
+ @Override
+ public void skipFloats(int total) {
+
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+
+ }
+
+ @Override
+ public void skipBinary(int total) {
Review comment:
I left this file as is from the previous PR. For the moment I'll just call skip from the underlying non-vectorized version, then implement the vectorized version in a subsequent PR.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+
Review comment:
Fixed
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingIntegerSuite.scala
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForInteger
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.IntegerType
+
+/**
+ * Read tests for vectorized Delta binary packed Integer reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ */
+class ParquetDeltaEncodingIntegerSuite extends ParquetCompatibilityTest with SharedSparkSession {
Review comment:
I made an attempt to do so earlier. But I couldn't get it right :( . Also, this way it mirrors the Parquet library more exactly.
Anyway, I managed to get these merged. The result may not be pretty :(
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
Review comment:
I think we do. The original unit tests have interleaving read and skip. To continue to read after a skip, we need to have read the previous value.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
Review comment:
Oh dear. I implemented only the methods the original PR had implemented. On closer look we also need support for byte, short, date, timestamp, yearmonth interval, and daytime interval datatypes which are stored as int32 or int64.
Perf note: Rebased dates and timestamps appear to be a backward compatibility fix and incur the penalty of checking if the value needs to be rebased.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
+ }
+
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
Review comment:
But not for long (there's a horrible pun in here somewhere).
I need this for the vectorized implemenatation of DeltaByteArrayReader (which I did not include to make review easier).
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966762517
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49596/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-962042625
Thank you for the review @sunchao! Let me address the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r744037277
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -280,6 +277,20 @@ private void initDataReader(
}
}
+ private ValuesReader getValuesReader(Encoding encoding) {
+ switch (encoding) {
Review comment:
Could you use `two-space` indentation like the other part of this file, @parthchandra ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999936409
**[Test build #146491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146491/testReport)** for PR 34471 at commit [`64cc82f`](https://github.com/apache/spark/commit/64cc82fc2898cf553900b589f93bdd4a5887060d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972529729
I also checked on a real-world dataset. After writing the data compression was marginally better.
Parquet v1 (RLE) = 301.9 GB, Parquet v2 (DELTA_BINARY, DELTA_BYTE_ARRAY) = 299.9 GB. The bulk of the data was String data so this does not reflect the effectiveness of DELTA_BINARY since most of the data would be encoded using DELTA_BYTE_ARRAY.
Haven't been able to run a read benchmark on this yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966698908
**[Test build #145121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145121/testReport)** for PR 34471 at commit [`3572eaa`](https://github.com/apache/spark/commit/3572eaa42ce5360cc69b92888383bb39bef13a01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994202952
**[Test build #146200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146200/testReport)** for PR 34471 at commit [`f472135`](https://github.com/apache/spark/commit/f472135a04a3129914b94ddddc03d506d07626a1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r768089566
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingSuite.scala
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.{DeltaBinaryPackingValuesWriterForInteger, DeltaBinaryPackingValuesWriterForLong}
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{IntegerType, IntegralType, LongType}
+
+/**
+ * Read tests for vectorized Delta binary packed reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForLongTest
+ */
+abstract class ParquetDeltaEncodingSuite[T] extends ParquetCompatibilityTest
+ with SharedSparkSession {
+ protected var blockSize = 128
+ protected var miniBlockNum = 4
+ protected var reader: VectorizedDeltaBinaryPackedReader = _
+ protected var writableColumnVector: WritableColumnVector = _
+ protected var writer: ValuesWriter = _
+ protected var random: Random = _
+
+ protected def getSparkSqlType: IntegralType
+
+ protected def writeData(data: Array[T]): Unit
+
+ protected def writeData(data: Array[T], length: Int): Unit
+
+ protected def initValuesWriter (): Unit
+
+// protected def allocArrayOfType[T: ClassTag](size: Int): Array[T] = new Array[T](size)
Review comment:
nit: remove?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.catalyst.util.RebaseDateTime;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface. DELTA_BINARY_PACKED is a delta encoding for integer and long types that stores values
+ * as a delta between consecutive values. Delta values are themselves bit packed. Similar to RLE but
+ * is more effective in the case of large variation of values in the encoded column.
+ * <p>
+ * DELTA_BINARY_PACKED is the default encoding for integer and long columns in Parquet V2.
+ * <p>
+ * Supported Types: INT32, INT64
+ * <p>
+ *
+ * @see <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5">
+ * Parquet format encodings: DELTA_BINARY_PACKED</a>
+ */
+public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ // variables to keep state of the current block and miniblock
+ private long lastValueRead; // needed to compute the next value
+ private long minDeltaInCurrentBlock; // needed to compute the next value
+ // currentMiniBlock keeps track of the mini block within the current block that
+ // we read and decoded most recently. Only used as an index into
+ // bitWidths array
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniBlock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ // temporary buffers used by readByte, readShort, readInteger, and readLong
+ byte byteVal;
+ short shortVal;
+ int intVal;
+ long longVal;
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+ }
+
+ @Override
+ public byte readByte() {
+ readValues(1, null, 0, (w, r, v) -> byteVal = (byte) v);
+ return byteVal;
+ }
+
+ @Override
+ public short readShort() {
+ readValues(1, null, 0, (w, r, v) -> shortVal = (short) v);
+ return shortVal;
+ }
+
+ @Override
+ public int readInteger() {
+ readValues(1, null, 0, (w, r, v) -> intVal = (int) v);
+ return intVal;
+ }
+
+ @Override
+ public long readLong() {
+ readValues(1, null, 0, (w, r, v) -> longVal = v);
+ return longVal;
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putByte(r, (byte) v));
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putShort(r, (short) v));
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> w.putInt(r, (int) v));
+ }
+
+ // Based on VectorizedPlainValuesReader.readIntegersWithRebase
+ @Override
+ public final void readIntegersWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianDay()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putInt(r, RebaseDateTime.rebaseJulianToGregorianDays((int) v));
+ }
+ } else {
+ w.putInt(r, (int) v);
+ }
+ });
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putLong(r, Integer.toUnsignedLong((int) v));
+ });
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ w.putByteArray(r, new BigInteger(Long.toUnsignedString(v)).toByteArray());
+ });
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public final void readLongsWithRebase(
+ int total, WritableColumnVector c, int rowId, boolean failIfRebase) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ if (v < RebaseDateTime.lastSwitchJulianTs()) {
+ if (failIfRebase) {
+ throw DataSourceUtils.newRebaseExceptionInRead("Parquet");
+ } else {
+ w.putLong(r, RebaseDateTime.rebaseJulianToGregorianMicros(v));
+ }
+ } else {
+ w.putLong(r, v);
+ }
+ });
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ skipValues(total);
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ skipValues(total);
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "No more values to read, total value count is " + valuesRead);
Review comment:
nit: maybe revise this message a bit, since "total value count is + valuesRead" looks a bit confusing.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingSuite.scala
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.{DeltaBinaryPackingValuesWriterForInteger, DeltaBinaryPackingValuesWriterForLong}
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{IntegerType, IntegralType, LongType}
+
+/**
+ * Read tests for vectorized Delta binary packed reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForLongTest
+ */
+abstract class ParquetDeltaEncodingSuite[T] extends ParquetCompatibilityTest
+ with SharedSparkSession {
+ protected var blockSize = 128
+ protected var miniBlockNum = 4
+ protected var reader: VectorizedDeltaBinaryPackedReader = _
+ protected var writableColumnVector: WritableColumnVector = _
+ protected var writer: ValuesWriter = _
+ protected var random: Random = _
+
+ protected def getSparkSqlType: IntegralType
+
+ protected def writeData(data: Array[T]): Unit
+
+ protected def writeData(data: Array[T], length: Int): Unit
+
+ protected def initValuesWriter (): Unit
+
+// protected def allocArrayOfType[T: ClassTag](size: Int): Array[T] = new Array[T](size)
+
+ protected def allocDataArray(size: Int): Array[T]
+
+ protected def getNextRandom: T
+
+ protected def getTypeMinValue: T
+
+ protected def getTypeMaxValue: T
+
+ protected def readData(total: Int, columnVector : WritableColumnVector, rowId: Int): Unit
+
+ protected def skip(total: Int): Unit
+
+ protected def readDataFromVector(columnVector : WritableColumnVector, rowId: Int): T
+
+ protected def estimatedSize(length: Int) : Double
+
+ protected def setValue(arr: Array[T], index: Int, value: Int): Unit
+ protected def compareValues(expected: Int, actual: T) : Boolean
+
+
+ protected override def beforeEach(): Unit = {
+ blockSize = 128
+ miniBlockNum = 4
+ random = new Random(0)
+ initValuesWriter()
+ super.beforeAll()
+ }
+
+ test("read when data is aligned with block") {
+ val data = allocDataArray(5 * blockSize)
+ for (i <- 0 until blockSize * 5) {
+ data(i) = getNextRandom
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when block is not fully written") {
+ val data = allocDataArray(blockSize - 3)
+ for (i <- data.indices) {
+ data(i) = getNextRandom
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when mini block is not fully written") {
+ val miniBlockSize = blockSize / miniBlockNum
+ val data = allocDataArray(miniBlockSize - 3)
+ for (i <- data.indices) {
+ data(i) = getNextRandom
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read with negative deltas") {
+ val data = allocDataArray(blockSize)
+ for (i <- data.indices) {
+ setValue(data, i, 10 - (i * 32 - random.nextInt(6)))
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when deltas are same") {
+ val data = allocDataArray(2 * blockSize)
+ for (i <- 0 until blockSize) {
+ setValue(data, i, i * 32)
+ }
+ for (i <- blockSize until 2 * blockSize) {
+ setValue(data, i, 0)
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when values are same") {
+ val data = allocDataArray(2 * blockSize)
+ for (i <- 0 until blockSize) {
+ setValue(data, i, 3)
+ }
+ for (i <- blockSize until 2 * blockSize) {
+ setValue(data, i, 0)
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when delta is 0 for each block") {
+ val data = allocDataArray(5 * blockSize + 1)
+ for (i <- data.indices) {
+ setValue(data, i, (i - 1) / blockSize)
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read when data is not aligned with block") {
+ val data = allocDataArray(5 * blockSize + 3)
+ for (i <- data.indices) {
+ setValue(data, i, random.nextInt(20) - 10)
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("read max min value") {
+ val data = allocDataArray(10)
+ for (i <- data.indices) {
+ if (i % 2 == 0) data(i) = getTypeMinValue
+ else data(i) = getTypeMaxValue
+ }
+ shouldWriteAndRead(data)
+ }
+
+ test("throw exception when read more than written") {
+ val data = allocDataArray(5 * blockSize + 1)
+ for (i <- data.indices) {
+ setValue(data, i, i * 32)
+ }
+ shouldWriteAndRead(data)
+ try readData(1, writableColumnVector, data.length)
+ catch {
+ case e: ParquetDecodingException =>
+ assert("no more values to read, total value count is " + data.length == e.getMessage)
+ }
+ }
+
+ test("skip()") {
+ val data = allocDataArray(5 * blockSize + 1)
+ for (i <- data.indices) {
+ setValue(data, i, i * 32)
+ }
+ writeData(data)
+ reader = new VectorizedDeltaBinaryPackedReader
+ reader.initFromPage(100, writer.getBytes.toInputStream)
+ writableColumnVector = new OnHeapColumnVector(data.length, getSparkSqlType)
+ for (i <- data.indices) {
+ if (i % 3 == 0) {
+ skip(1)
+ } else {
+ readData(1, writableColumnVector, i)
+ assert(compareValues(i * 32, readDataFromVector(writableColumnVector, i)))
+ }
+ }
+ }
+
+ test("SkipN()") {
+ val data = allocDataArray(5 * blockSize + 1)
+ for (i <- data.indices) {
+ setValue(data, i, i * 32)
+ }
+ writeData(data)
+ reader = new VectorizedDeltaBinaryPackedReader
+ reader.initFromPage(100, writer.getBytes.toInputStream)
+ writableColumnVector = new OnHeapColumnVector(data.length, getSparkSqlType)
+ var skipCount = 0
+ var i = 0
+ while (i < data.length) {
+ skipCount = (data.length - i) / 2
+ readData(1, writableColumnVector, i)
+ assert(compareValues(i * 32, readDataFromVector(writableColumnVector, i)))
+ skip(skipCount)
+
+ i += skipCount + 1
+ }
+ }
+
+ test("random data test") {
+ val maxSize = 1000
+ val data = allocDataArray(maxSize)
+ for (round <- 0 until 100000) {
+ val size = random.nextInt(maxSize)
+ for (i <- 0 until size) {
+ data(i) = getNextRandom
+ }
+ shouldReadAndWrite(data, size)
+ writer.reset()
+ }
+ }
+
+ @throws[IOException]
+ private def shouldWriteAndRead(data: Array[T]): Unit = {
+ shouldReadAndWrite(data, data.length)
+ }
+
+ private def shouldReadAndWrite(data: Array[T], length: Int): Unit = {
+ writeData(data, length)
+ reader = new VectorizedDeltaBinaryPackedReader
+ val page = writer.getBytes.toByteArray
+
+ assert(estimatedSize(length) >= page.length)
+ writableColumnVector = new OnHeapColumnVector(data.length, getSparkSqlType)
+ reader.initFromPage(100, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)))
+ readData(length, writableColumnVector, 0)
+ for (i <- 0 until length) {
+ assert(data(i) == readDataFromVector(writableColumnVector, i))
+ }
+ }
+
+}
+
+class ParquetDeltaEncodingInteger extends ParquetDeltaEncodingSuite[Int] {
+
+ override protected def getSparkSqlType: IntegralType = IntegerType
+ override protected def writeData(data: Array[Int]): Unit = writeData(data, data.length)
+
+ override protected def writeData(data: Array[Int], length: Int): Unit = {
+ for (i <- 0 until length) {
+ writer.writeInteger(data(i))
+ }
+ }
+
+ override protected def initValuesWriter (): Unit = {
+ writer = new DeltaBinaryPackingValuesWriterForInteger(
+ blockSize,
+ miniBlockNum,
+ 100,
+ 200,
+ new DirectByteBufferAllocator())
+ }
+
+ override protected def allocDataArray(size: Int): Array[Int] = new Array[Int](size)
+
+ override protected def getNextRandom: Int = random.nextInt
+ override protected def getTypeMinValue: Int = Int.MinValue
+ override protected def getTypeMaxValue: Int = Int.MaxValue
+
+ override protected def readData(total: Int, columnVector : WritableColumnVector, rowId: Int): Unit
+ = reader.readIntegers(total, columnVector, rowId)
+
+ override protected def skip(total: Int): Unit = reader.skipIntegers(total)
+
+ override protected def readDataFromVector(columnVector: WritableColumnVector, rowId: Int): Int =
+ columnVector.getInt(rowId)
+
+ override protected def estimatedSize(length: Int) : Double = {
+ val miniBlockSize = blockSize / miniBlockNum
+ val miniBlockFlushed = Math.ceil((length.toDouble - 1) / miniBlockSize)
+ val blockFlushed = Math.ceil((length.toDouble - 1) / blockSize)
+ 4 * 5 /* blockHeader */ +
+ 4 * miniBlockFlushed * miniBlockSize /* data(aligned to miniBlock) */ +
+ blockFlushed * miniBlockNum /* bitWidth of mini blocks */ +
+ (5.0 * blockFlushed) /* min delta for each block */
+ }
+
+ override protected def setValue(arr: Array[Int], index: Int, value: Int): Unit =
+ arr(index) = value
+
+ override protected def compareValues(expected: Int, actual: Int) : Boolean =
+ expected == actual
+
+}
+
+class ParquetDeltaEncodingLong extends ParquetDeltaEncodingSuite[Long] {
+
+ override protected def getSparkSqlType: IntegralType = LongType
+ override protected def writeData(data: Array[Long]): Unit = writeData(data, data.length)
+
+ override protected def writeData(data: Array[Long], length: Int): Unit = {
+ for (i <- 0 until length) {
+ writer.writeLong(data(i))
+ }
+ }
+
+ override protected def initValuesWriter (): Unit = {
+ writer = new DeltaBinaryPackingValuesWriterForLong(
+ blockSize,
+ miniBlockNum,
+ 100,
+ 200,
+ new DirectByteBufferAllocator())
+ }
+
+ override protected def allocDataArray(size: Int): Array[Long] = new Array[Long](size)
+
+ override protected def getNextRandom: Long = random.nextLong
+ override protected def getTypeMinValue: Long = Long.MinValue
+ override protected def getTypeMaxValue: Long = Long.MaxValue
+
+ override protected def readData(total: Int, columnVector: WritableColumnVector, rowId: Int): Unit
+ = reader.readLongs(total, columnVector, rowId)
Review comment:
nit: indentation
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -142,13 +154,23 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
}
}
+ sqlBenchmark.addCase("SQL Parquet Vectorized (Delta Binary)") { _ =>
+ spark.sql("select sum(id) from parquetV2Table").noop()
+ }
+
+ sqlBenchmark.addCase("SQL Parquet MR (Delta Binary)") { _ =>
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ spark.sql("select sum(id) from parquetV2Table").noop()
+ }
+ }
+
sqlBenchmark.addCase("SQL ORC Vectorized") { _ =>
- spark.sql(s"SELECT $query FROM orcTable").noop()
+ spark.sql("SELECT sum(id) FROM orcTable").noop()
Review comment:
hmm why this change?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -147,6 +159,16 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
}
}
+ sqlBenchmark.addCase("SQL Parquet Vectorized (Delta Binary)") { _ =>
Review comment:
This won't work yet because of the `BooleanType` added recently.
Error:
```
[error] Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'sum(parquetv2table.id)' due to data type mismatch: function sum requires numeric or interval types, not boolean; line 1 pos 7;
[error] 'Aggregate [unresolvedalias(sum(id#40), None)]
[error] +- SubqueryAlias parquetv2table
[error] +- View (`parquetV2Table`, [id#40])
[error] +- Relation [id#40] parquet
```
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingSuite.scala
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.{DeltaBinaryPackingValuesWriterForInteger, DeltaBinaryPackingValuesWriterForLong}
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{IntegerType, IntegralType, LongType}
+
+/**
+ * Read tests for vectorized Delta binary packed reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForLongTest
+ */
+abstract class ParquetDeltaEncodingSuite[T] extends ParquetCompatibilityTest
+ with SharedSparkSession {
+ protected var blockSize = 128
+ protected var miniBlockNum = 4
+ protected var reader: VectorizedDeltaBinaryPackedReader = _
+ protected var writableColumnVector: WritableColumnVector = _
+ protected var writer: ValuesWriter = _
+ protected var random: Random = _
+
+ protected def getSparkSqlType: IntegralType
+
+ protected def writeData(data: Array[T]): Unit
+
+ protected def writeData(data: Array[T], length: Int): Unit
+
+ protected def initValuesWriter (): Unit
+
+// protected def allocArrayOfType[T: ClassTag](size: Int): Array[T] = new Array[T](size)
+
+ protected def allocDataArray(size: Int): Array[T]
+
+ protected def getNextRandom: T
+
+ protected def getTypeMinValue: T
+
+ protected def getTypeMaxValue: T
+
+ protected def readData(total: Int, columnVector : WritableColumnVector, rowId: Int): Unit
+
+ protected def skip(total: Int): Unit
+
+ protected def readDataFromVector(columnVector : WritableColumnVector, rowId: Int): T
+
+ protected def estimatedSize(length: Int) : Double
+
+ protected def setValue(arr: Array[T], index: Int, value: Int): Unit
+ protected def compareValues(expected: Int, actual: T) : Boolean
+
+
+ protected override def beforeEach(): Unit = {
+ blockSize = 128
Review comment:
nit: these seem redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-998175819
@parthchandra could you address the unit tests failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966747874
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49593/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966825182
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959699641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958547677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999763026
Thank you @sunchao. I had gone thru the log and failed to see the test(s) that had failed. One of the unit tests was checking for the error message that accompanied an exception and as part of the review I had changed the error message! Updated the test. The tests should pass now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999823617
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50967/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999820120
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50967/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-961492262
cc @sadikovi @viirya @dongjoon-hyun too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958404969
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49330/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959709152
**[Test build #144880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144880/testReport)** for PR 34471 at commit [`10659d3`](https://github.com/apache/spark/commit/10659d3bc9c1f3b3b9f8c29000f9f75e95281777).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972531356
**[Test build #145367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145367/testReport)** for PR 34471 at commit [`2574e69`](https://github.com/apache/spark/commit/2574e69577285075f102d7a39ef373ced8bdaaea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958606731
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144864/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958490904
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49330/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958557735
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144861/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-962042625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959805996
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49350/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958531680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966784677
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49596/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966807834
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966715846
**[Test build #145122 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145122/testReport)** for PR 34471 at commit [`6021d90`](https://github.com/apache/spark/commit/6021d90a2a6bd9a90724bd30b4cffc999330f096).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966731149
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49593/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966761997
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966825182
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r768903287
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -147,6 +159,16 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
}
}
+ sqlBenchmark.addCase("SQL Parquet Vectorized (Delta Binary)") { _ =>
Review comment:
Right, Boolean is encoded as RLE in Parquet format V2, so we'd need to handle it too. I think we should raise a separate PR for this.
For now, can you remove the changes in `DataSourceReadBenchmark` and split it out into a separate PR later once we have the boolean type supported?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994197850
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50674/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r780024037
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -76,6 +79,7 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
saveAsCsvTable(testDf, dir.getCanonicalPath + "/csv")
saveAsJsonTable(testDf, dir.getCanonicalPath + "/json")
saveAsParquetTable(testDf, dir.getCanonicalPath + "/parquet")
+ saveAsParquetV2Table(testDf, dir.getCanonicalPath + "/parquetV2")
Review comment:
I found that there are still unsupported encoding in Data Page V2, such as RLE for Boolean. It seems that it is not time to update the benchmark, please ignore my previous comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r780024037
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
##########
@@ -76,6 +79,7 @@ object DataSourceReadBenchmark extends SqlBasedBenchmark {
saveAsCsvTable(testDf, dir.getCanonicalPath + "/csv")
saveAsJsonTable(testDf, dir.getCanonicalPath + "/json")
saveAsParquetTable(testDf, dir.getCanonicalPath + "/parquet")
+ saveAsParquetV2Table(testDf, dir.getCanonicalPath + "/parquetV2")
Review comment:
I found that there are still unsupported encoding in Data Page V2, such as RLE for Boolean. It seems that it is not time to update the benchmark, please ignore my previous comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994197850
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50674/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-994195217
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50674/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966759783
**[Test build #145121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145121/testReport)** for PR 34471 at commit [`3572eaa`](https://github.com/apache/spark/commit/3572eaa42ce5360cc69b92888383bb39bef13a01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class VectorizedDeltaBinaryPackedReader extends VectorizedReaderBase `
* `public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase `
* `public class VectorizedReaderBase extends ValuesReader implements VectorizedValuesReader `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966779056
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49596/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966715846
**[Test build #145122 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145122/testReport)** for PR 34471 at commit [`6021d90`](https://github.com/apache/spark/commit/6021d90a2a6bd9a90724bd30b4cffc999330f096).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966737754
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49592/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-966737754
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49592/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-972692219
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145367/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959871361
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144880/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958599302
**[Test build #144864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144864/testReport)** for PR 34471 at commit [`10659d3`](https://github.com/apache/spark/commit/10659d3bc9c1f3b3b9f8c29000f9f75e95281777).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL]Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-958302148
**[Test build #144861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144861/testReport)** for PR 34471 at commit [`fc9683b`](https://github.com/apache/spark/commit/fc9683bfc0af15050ddfdeef50a9ff7a45078bb8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-959849943
**[Test build #144880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144880/testReport)** for PR 34471 at commit [`10659d3`](https://github.com/apache/spark/commit/10659d3bc9c1f3b3b9f8c29000f9f75e95281777).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34471:
URL: https://github.com/apache/spark/pull/34471#discussion_r743047058
##########
File path: .gitignore
##########
@@ -33,6 +33,7 @@ R/pkg/tests/fulltests/Rplots.pdf
build/*.jar
build/apache-maven*
build/scala*
+build/zinc*
Review comment:
nit: unrelated change?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
+ }
+
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
Review comment:
this doesn't seem to be used anywhere
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
Review comment:
maybe add some comments to these fields - it's a bit confusing that `currentMiniBlock` is an integer
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ for (int i = 0; i < total; i++) {
+ Binary binary = deltaByteArrayReader.readBytes();
+ ByteBuffer buffer = binary.toByteBuffer();
+ if (buffer.hasArray()) {
+ c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
+ binary.length());
+ } else {
+ byte[] bytes = new byte[binary.length()];
+ buffer.get(bytes);
+ c.putByteArray(rowId + i, bytes);
+ }
+ }
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+
+ }
+
+ @Override
+ public void skipBytes(int total) {
+
+ }
+
+ @Override
+ public void skipShorts(int total) {
+
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+
+ }
+
+ @Override
+ public void skipLongs(int total) {
+
+ }
+
+ @Override
+ public void skipFloats(int total) {
+
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+
+ }
+
+ @Override
+ public void skipBinary(int total) {
Review comment:
we need this too
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -166,7 +166,7 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
readState.resetForNewPage(pageValueCount, pageFirstRowIndex);
}
PrimitiveType.PrimitiveTypeName typeName =
- descriptor.getPrimitiveType().getPrimitiveTypeName();
+ descriptor.getPrimitiveType().getPrimitiveTypeName();
Review comment:
ditto
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
Review comment:
seems it's better to have an abstract class inheriting `ValuesReader` and `VectorizedValuesReader` with this default behavior defined, rather than repeating the same thing in all the different value readers.
this can be done separately though.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
Review comment:
hmm I wonder why we decrement `remaining` here.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
Review comment:
should this be `w.putInt`?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -63,4 +64,36 @@
void skipDoubles(int total);
void skipBinary(int total);
void skipFixedLenByteArray(int total, int len);
+
+ /**
+ * An interface to write columnar output in various ways
+ */
+ @FunctionalInterface
+ interface IntegerOutputWriter {
+ void write(WritableColumnVector c, int rowId, long val);
Review comment:
maybe add some comments for this? what are `c`, `rowId` and `val` for?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+
Review comment:
ditto
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
Review comment:
nit: add space after `//`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
+ BytePackerForLong packer = Packer.LITTLE_ENDIAN.newBytePackerForLong(
+ bitWidths[currentMiniBlock]);
+ for (int j = 0; j < miniBlockSizeInValues; j += 8) {
+ ByteBuffer buffer = in.slice(packer.getBitWidth());
+ packer.unpack8Values(buffer, buffer.position(), unpackedValuesBuffer, j);
Review comment:
wonder if we can unpack 32 values at once, since the [spec](https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5) says the number of values in a mini block is a multiple of 32.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
Review comment:
nit: remove `/*unused*/`: it's used although just in precondition check
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
Review comment:
maybe add a bit more information, like when it will be used, what data types it supports, etc
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
+ outputWriter.write(c, rowId, firstValue);
+ lastValueRead = firstValue;
+ rowId++;
+ remaining--;
+ }
+ while (remaining > 0) {
+ int n;
+ try {
+ n = loadMiniBlockToOutput(remaining, c, rowId, outputWriter);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("Error reading mini block.", e);
+ }
+ rowId += n;
+ remaining -= n;
+ }
+ valuesRead = total - remaining;
+ }
+
+
+ /**
+ * Read from a mini block. Read at most 'remaining' values into output.
+ *
+ * @return the number of values read into output
+ */
+ private int loadMiniBlockToOutput(int remaining, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) throws IOException {
+
+ // new block; read the block header
+ if (remainingInBlock == 0) {
+ readBlockHeader();
+ }
+
+ // new miniblock, unpack the miniblock
+ if (remainingInMiniBlock == 0) {
+ unpackMiniBlock();
+ }
+
+ //read values from miniblock
+ int valuesRead = 0;
+ for (int i = miniBlockSizeInValues - remainingInMiniBlock;
+ i < miniBlockSizeInValues && valuesRead < remaining; i++) {
+ //calculate values from deltas unpacked for current block
+ long outValue = lastValueRead + minDeltaInCurrentBlock + unpackedValuesBuffer[i];
+ lastValueRead = outValue;
+ outputWriter.write(c, rowId + valuesRead, outValue);
+ remaining--;
+ remainingInBlock--;
+ remainingInMiniBlock--;
+ valuesRead++;
+ }
+
+ return valuesRead;
+ }
+
+ private void readBlockHeader() {
+ try {
+ minDeltaInCurrentBlock = BytesUtils.readZigZagVarLong(in);
+ } catch (IOException e) {
+ throw new ParquetDecodingException("can not read min delta in current block", e);
+ }
+ readBitWidthsForMiniBlocks();
+ remainingInBlock = blockSizeInValues;
+ currentMiniBlock = 0;
+ remainingInMiniBlock = 0;
+ }
+
+ /**
+ * mini block has a size of 8*n, unpack 8 value each time
+ * @see org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesReader#unpackMiniBlock
+ */
+ private void unpackMiniBlock() throws IOException {
+ this.unpackedValuesBuffer = new long[miniBlockSizeInValues];
Review comment:
maybe we can reuse `this.unpackedValuesBuffer`?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -122,4 +126,40 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
}
}
}
+
+ test("parquet v2 pages - delta encoding") {
+ val extraOptions = Map[String, String](
+ ParquetOutputFormat.WRITER_VERSION -> ParquetProperties.WriterVersion.PARQUET_2_0.toString,
+ ParquetOutputFormat.ENABLE_DICTIONARY -> "false"
+ )
+
+ val hadoopConf = spark.sessionState.newHadoopConfWithOptions(extraOptions)
+ withSQLConf(
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",
+ ParquetOutputFormat.JOB_SUMMARY_LEVEL -> "ALL") {
Review comment:
why we need this?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingIntegerSuite.scala
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.IOException
+import java.nio.ByteBuffer
+import java.util.Random
+
+import org.apache.parquet.bytes.{ByteBufferInputStream, DirectByteBufferAllocator}
+import org.apache.parquet.column.values.ValuesWriter
+import org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForInteger
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.IntegerType
+
+/**
+ * Read tests for vectorized Delta binary packed Integer reader.
+ * Translated from
+ * org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForIntegerTest
+ */
+class ParquetDeltaEncodingIntegerSuite extends ParquetCompatibilityTest with SharedSparkSession {
Review comment:
can we combine this with `ParquetDeltaEncodingLongSuite`? we can have a base class with the common code (need to abstract a few things like how to allocate an array, how to generate a rand int/long, and Parquet's value writer class etc.)
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ */
+public class VectorizedDeltaByteArrayReader extends ValuesReader implements VectorizedValuesReader {
+ private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ deltaByteArrayReader.initFromPage(valueCount, in);
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ return 0;
Review comment:
why return 0 here?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
Review comment:
I think we'll need to implement these too.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
Review comment:
I think this can be done more efficiently, for instance we don't need to unpack the bits anymore, and don't need to compute the original value from delta, etc.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+import java.io.IOException;
+
+/**
+ * An implementation of the Parquet DELTA_BINARY_PACKED decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaBinaryPackedReader extends ValuesReader
+ implements VectorizedValuesReader {
+
+ // header data
+ private int blockSizeInValues;
+ private int miniBlockNumInABlock;
+ private int totalValueCount;
+ private long firstValue;
+
+ private int miniBlockSizeInValues;
+
+ // values read by the caller
+ private int valuesRead = 0;
+
+ //variables to keep state of the current block and miniblock
+ private long lastValueRead;
+ private long minDeltaInCurrentBlock;
+ private int currentMiniBlock = 0;
+ private int[] bitWidths; // bit widths for each miniblock in the current block
+ private int remainingInBlock = 0; // values in current block still to be read
+ private int remainingInMiniBlock = 0; // values in current mini block still to be read
+ private long[] unpackedValuesBuffer;
+
+ private ByteBufferInputStream in;
+
+ @SuppressWarnings("unused")
+ @Override
+ public void initFromPage(/*unused*/int valueCount, ByteBufferInputStream in) throws IOException {
+ Preconditions.checkArgument(valueCount >= 1,
+ "Page must have at least one value, but it has " + valueCount);
+ this.in = in;
+
+ // Read the header
+ this.blockSizeInValues = BytesUtils.readUnsignedVarInt(in);
+ this.miniBlockNumInABlock = BytesUtils.readUnsignedVarInt(in);
+ double miniSize = (double) blockSizeInValues / miniBlockNumInABlock;
+ Preconditions.checkArgument(miniSize % 8 == 0,
+ "miniBlockSize must be multiple of 8, but it's " + miniSize);
+ this.miniBlockSizeInValues = (int) miniSize;
+ this.totalValueCount = BytesUtils.readUnsignedVarInt(in);
+ this.bitWidths = new int[miniBlockNumInABlock];
+
+ // read the first value
+ firstValue = BytesUtils.readZigZagVarLong(in);
+
+ }
+
+ @Override
+ public void skip() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte readByte() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public short readShort() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Binary readBinary(int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBooleans(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBytes(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readShorts(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readIntegers(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, (w, r, v) -> {
+ c.putInt(r, (int) v);
+ });
+ }
+
+ @Override
+ public void readIntegersWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException("Only readIntegers is valid.");
+ }
+
+ @Override
+ public void readUnsignedIntegers(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readUnsignedLongs(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readLongs(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, WritableColumnVector::putLong);
+ }
+
+ @Override
+ public void readLongsWithRebase(int total, WritableColumnVector c, int rowId,
+ boolean failIfRebase) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readFloats(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readDoubles(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBooleans(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBytes(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipShorts(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipIntegers(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipLongs(int total) {
+ // Read the values but don't write them out (the writer output method is a no-op)
+ readValues(total, null, -1, (w, r, v) -> {
+ });
+ }
+
+ @Override
+ public void skipFloats(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipDoubles(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void skipFixedLenByteArray(int total, int len) {
+ throw new UnsupportedOperationException();
+ }
+
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ IntegerOutputWriter outputWriter) {
+ int remaining = total;
+ if (valuesRead + total > totalValueCount) {
+ throw new ParquetDecodingException(
+ "no more values to read, total value count is " + valuesRead);
+ }
+ // First value
+ if (valuesRead == 0) {
+ //c.putInt(rowId, (int)firstValue);
Review comment:
nit: remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999945283
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146491/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999823617
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50967/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-999764680
**[Test build #146491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146491/testReport)** for PR 34471 at commit [`64cc82f`](https://github.com/apache/spark/commit/64cc82fc2898cf553900b589f93bdd4a5887060d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34471: [SPARK-36879][SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34471:
URL: https://github.com/apache/spark/pull/34471#issuecomment-992741489
**[Test build #146144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146144/testReport)** for PR 34471 at commit [`91f7534`](https://github.com/apache/spark/commit/91f7534c87c67e6e456a06f6f250dd63e8b14f3c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org