You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/20 18:50:51 UTC
[GitHub] [spark] parthchandra opened a new pull request #35262: [SPARK-36879][SQL][FOLLOWUP] Support Parquet v2 data page encodings for the vectorized path
parthchandra opened a new pull request #35262:
URL: https://github.com/apache/spark/pull/35262
### What changes were proposed in this pull request?
This PR provides a vectorized implementation of the DELTA_BYTE_ARRAY encoding of Parquet V2. The PR also implements the DELTA_LENGTH_BYTE_ARRAY encoding which is needed by the former.
### Why are the changes needed?
The current support for Parquet V2 in the vectorized reader uses a non-vectorized version of the above encoding and needs to be vectorized.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Reproduces all the tests for the encodings from the Parquet implementation. Also adds more cases to the Parquet Encoding test suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1029295384
Thank you for your explanation @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r802122170
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,7 +180,7 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
readState.resetForNewPage(pageValueCount, pageFirstRowIndex);
}
PrimitiveType.PrimitiveTypeName typeName =
- descriptor.getPrimitiveType().getPrimitiveTypeName();
+ descriptor.getPrimitiveType().getPrimitiveTypeName();
Review comment:
nit: unrelated change
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -295,13 +307,13 @@ private ValuesReader getValuesReader(Encoding encoding) {
return new VectorizedDeltaBinaryPackedReader();
case RLE:
PrimitiveType.PrimitiveTypeName typeName =
- this.descriptor.getPrimitiveType().getPrimitiveTypeName();
+ this.descriptor.getPrimitiveType().getPrimitiveTypeName();
Review comment:
ditto
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
Review comment:
Should we use `prefixLengthReader.getTotalValueCount()` here? `valueCount` includes nulls while here we're only reading non-null values right?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -295,13 +307,13 @@ private ValuesReader getValuesReader(Encoding encoding) {
return new VectorizedDeltaBinaryPackedReader();
case RLE:
PrimitiveType.PrimitiveTypeName typeName =
- this.descriptor.getPrimitiveType().getPrimitiveTypeName();
+ this.descriptor.getPrimitiveType().getPrimitiveTypeName();
// RLE encoding only supports boolean type `Values`, and `bitwidth` is always 1.
if (typeName == BOOLEAN) {
return new VectorizedRleValuesReader(1);
} else {
throw new UnsupportedOperationException(
- "RLE encoding is not supported for values of type: " + typeName);
+ "RLE encoding is not supported for values of type: " + typeName);
Review comment:
ditto
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
Review comment:
Should we read `SQLConf.offHeapColumnVectorEnabled` and use `OffHeapColumnVector` if it's on?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
Review comment:
I'm not sure whether this will ever happen. The caller of `readValues` always ensure that `total > 0`. It's an implicit contract.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -69,6 +71,7 @@
protected MessageType fileSchema;
protected MessageType requestedSchema;
protected StructType sparkSchema;
+ protected ParsedVersion writerVersion;
Review comment:
Maybe it's worth adding some comments on why we track `writerVersion` here. It's a bit confusing when reading this.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
Review comment:
nit: maybe this comment belong to somewhere else? it's obvious that `int length = prefixLength + suffix.length;` doesn't copy bytes.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Hmm I think we should not skip writing if `length` is 0. Maybe:
```java
for (int i = 0; i < total; i++) {
length = lengthsVector.getInt(rowId + i);
try {
buffer = in.slice(length);
} catch (EOFException e) {
throw new ParquetDecodingException("Failed to read " + length + " bytes");
}
outputWriter.write(c, rowId + i, buffer, length);
}
```
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
Review comment:
nit: add space after `//`, also `getBinary` -> `readBinary`.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
Review comment:
nit: can we initialize this in the constructor? like `suffixReader`.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
Review comment:
We can also only eagerly decode prefix lengths in the constructor while only decode suffix here.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader() {
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
+ if (length > 0) {
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
+ }
+ }
+ currentRow += total;
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ if (total == 0) {
+ return;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(currentRow + i);
+ int remaining = length;
+ while (remaining > 0) {
+ remaining -= in.skip(length);
Review comment:
Should this be `remaining -= in.skip(remaining);`?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
+ int length = prefixLength + suffix.length;
+
+ // NOTE: due to PARQUET-246, it is important that we
Review comment:
nit: maybe we should move this block of comments to somewhere else. The following `if` statement is irrelevant to PARQUET-246
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
Review comment:
nit: init in ctor
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
+ int length = prefixLength + suffix.length;
+
+ // NOTE: due to PARQUET-246, it is important that we
+ // respect prefixLength which was read from prefixLengthReader,
+ // even for the *first* value of a page. Even though the first
+ // value of the page should have an empty prefix, it may not
+ // because of PARQUET-246.
+
+ // We have to do this to materialize the output
+ if (prefixLength != 0) {
+ // We could do
+ // c.putByteArray(rowId + i, previous, 0, prefixLength);
+ // c.putByteArray(rowId+i, suffix, prefixLength, suffix.length);
+ // previous = c.getBinary(rowId+1);
+ // but it incurs the same cost of copying the values twice _and_ c.getBinary
+ // is a _slow_ byte by byte copy
+ // The following always uses the faster system arraycopy method
+ byte[] out = new byte[length];
+ System.arraycopy(previous, 0, out, 0, prefixLength);
+ System.arraycopy(suffix, 0, out, prefixLength, suffix.length);
+ previous = out;
} else {
- byte[] bytes = new byte[binary.length()];
- buffer.get(bytes);
- c.putByteArray(rowId + i, bytes);
+ previous = suffix;
}
+ outputWriter.write(c, rowId + i, ByteBuffer.wrap(previous), previous.length);
+ currentRow++;
}
}
@Override
- public void skipBinary(int total) {
- for (int i = 0; i < total; i++) {
- deltaByteArrayReader.skip();
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, ByteBufferOutputWriter::writeArrayByteBuffer);
+ }
+
+ /**
+ * There was a bug (PARQUET-246) in which DeltaByteArrayWriter's reset() method did not clear the
+ * previous value state that it tracks internally. This resulted in the first value of all pages
Review comment:
In this case, what's the first value of `prefixLength`? is it 0 or the prefix length of the last value in the previous page?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
Review comment:
nit: do we need to init it with `new byte[0]`?
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 10153 10161 12 1.5 645.5 1.0X
+SQL Json 8463 8512 69 1.9 538.0 1.2X
+SQL Parquet Vectorized: DataPageV1 131 149 14 120.0 8.3 77.5X
+SQL Parquet Vectorized: DataPageV2 98 112 15 161.2 6.2 104.0X
+SQL Parquet MR: DataPageV1 1968 1968 0 8.0 125.1 5.2X
+SQL Parquet MR: DataPageV2 1735 1739 6 9.1 110.3 5.9X
+SQL ORC Vectorized 164 198 41 96.0 10.4 62.0X
+SQL ORC MR 1572 1581 12 10.0 100.0 6.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 102 107 14 154.6 6.5 1.0X
+ParquetReader Vectorized: DataPageV2 83 88 10 189.1 5.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 57 59 3 275.7 3.6 1.8X
+ParquetReader Vectorized -> Row: DataPageV2 38 40 3 416.3 2.4 2.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12263 12285 31 1.3 779.6 1.0X
+SQL Json 9495 9501 9 1.7 603.6 1.3X
+SQL Parquet Vectorized: DataPageV1 162 175 10 97.1 10.3 75.7X
+SQL Parquet Vectorized: DataPageV2 161 172 12 97.9 10.2 76.4X
+SQL Parquet MR: DataPageV1 2074 2105 44 7.6 131.9 5.9X
+SQL Parquet MR: DataPageV2 1974 1981 9 8.0 125.5 6.2X
+SQL ORC Vectorized 187 218 30 84.3 11.9 65.7X
+SQL ORC MR 1529 1553 34 10.3 97.2 8.0X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 205 214 12 76.8 13.0 1.0X
+ParquetReader Vectorized: DataPageV2 204 211 10 77.2 13.0 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 122 132 24 128.7 7.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 122 126 6 128.4 7.8 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12946 12955 12 1.2 823.1 1.0X
+SQL Json 9871 9876 8 1.6 627.6 1.3X
+SQL Parquet Vectorized: DataPageV1 157 200 34 100.0 10.0 82.3X
+SQL Parquet Vectorized: DataPageV2 229 242 14 68.8 14.5 56.7X
+SQL Parquet MR: DataPageV1 2388 2389 2 6.6 151.8 5.4X
+SQL Parquet MR: DataPageV2 2080 2087 10 7.6 132.2 6.2X
+SQL ORC Vectorized 240 285 23 65.6 15.2 54.0X
+SQL ORC MR 1699 1732 46 9.3 108.0 7.6X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 242 245 5 65.0 15.4 1.0X
+ParquetReader Vectorized: DataPageV2 309 314 9 50.9 19.7 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 227 268 18 69.3 14.4 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 294 312 25 53.5 18.7 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14331 14347 22 1.1 911.2 1.0X
+SQL Json 10406 10434 40 1.5 661.6 1.4X
+SQL Parquet Vectorized: DataPageV1 153 196 41 102.7 9.7 93.6X
+SQL Parquet Vectorized: DataPageV2 378 415 30 41.6 24.0 37.9X
+SQL Parquet MR: DataPageV1 2439 2446 11 6.4 155.1 5.9X
+SQL Parquet MR: DataPageV2 2181 2188 10 7.2 138.7 6.6X
+SQL ORC Vectorized 320 346 25 49.2 20.3 44.8X
+SQL ORC MR 1851 1853 3 8.5 117.7 7.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 258 262 9 60.9 16.4 1.0X
+ParquetReader Vectorized: DataPageV2 481 484 3 32.7 30.6 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 250 275 26 62.9 15.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 475 502 27 33.1 30.2 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 18450 18451 2 0.9 1173.0 1.0X
+SQL Json 12553 12562 13 1.3 798.1 1.5X
+SQL Parquet Vectorized: DataPageV1 259 272 12 60.8 16.5 71.3X
+SQL Parquet Vectorized: DataPageV2 534 566 22 29.4 34.0 34.5X
+SQL Parquet MR: DataPageV1 2529 2537 11 6.2 160.8 7.3X
+SQL Parquet MR: DataPageV2 2331 2334 4 6.7 148.2 7.9X
+SQL ORC Vectorized 424 460 36 37.1 27.0 43.5X
+SQL ORC MR 2009 2023 20 7.8 127.7 9.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 322 338 21 48.9 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 674 683 12 23.3 42.9 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 352 358 9 44.7 22.4 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 628 660 22 25.0 39.9 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14806 14816 14 1.1 941.3 1.0X
+SQL Json 11968 11969 1 1.3 760.9 1.2X
+SQL Parquet Vectorized: DataPageV1 150 184 26 105.0 9.5 98.8X
+SQL Parquet Vectorized: DataPageV2 147 183 32 107.2 9.3 100.9X
+SQL Parquet MR: DataPageV1 2338 2352 19 6.7 148.7 6.3X
+SQL Parquet MR: DataPageV2 2221 2267 65 7.1 141.2 6.7X
+SQL ORC Vectorized 475 494 29 33.1 30.2 31.1X
+SQL ORC MR 1967 1978 16 8.0 125.1 7.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 236 241 8 66.7 15.0 1.0X
+ParquetReader Vectorized: DataPageV2 237 241 9 66.3 15.1 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 218 244 25 72.1 13.9 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 218 251 21 72.2 13.8 1.1X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 19335 19346 15 0.8 1229.3 1.0X
+SQL Json 16112 16121 13 1.0 1024.4 1.2X
+SQL Parquet Vectorized: DataPageV1 257 278 29 61.1 16.4 75.1X
+SQL Parquet Vectorized: DataPageV2 258 268 9 60.9 16.4 74.9X
+SQL Parquet MR: DataPageV1 2542 2557 20 6.2 161.6 7.6X
+SQL Parquet MR: DataPageV2 2416 2439 32 6.5 153.6 8.0X
+SQL ORC Vectorized 593 605 18 26.5 37.7 32.6X
+SQL ORC MR 2134 2141 11 7.4 135.7 9.1X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 322 346 24 48.8 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 326 326 1 48.3 20.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 350 359 9 44.9 22.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 348 358 10 45.2 22.1 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 13899 14000 142 0.8 1325.5 1.0X
+SQL Json 11275 11289 20 0.9 1075.3 1.2X
+SQL Parquet Vectorized: DataPageV1 2092 2107 21 5.0 199.5 6.6X
+SQL Parquet Vectorized: DataPageV2 3073 3074 2 3.4 293.0 4.5X
+SQL Parquet MR: DataPageV1 4192 4212 29 2.5 399.8 3.3X
+SQL Parquet MR: DataPageV2 4133 4194 87 2.5 394.1 3.4X
+SQL ORC Vectorized 2218 2219 1 4.7 211.5 6.3X
+SQL ORC MR 3767 3776 12 2.8 359.3 3.7X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 7367 7387 28 1.4 702.6 1.0X
+SQL Json 6817 6817 0 1.5 650.1 1.1X
+SQL Parquet Vectorized: DataPageV1 602 618 15 17.4 57.5 12.2X
+SQL Parquet Vectorized: DataPageV2 599 610 15 17.5 57.1 12.3X
+SQL Parquet MR: DataPageV1 1888 1936 68 5.6 180.0 3.9X
+SQL Parquet MR: DataPageV2 2000 2018 25 5.2 190.7 3.7X
+SQL ORC Vectorized 527 545 22 19.9 50.2 14.0X
+SQL ORC MR 1916 1927 16 5.5 182.7 3.8X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 19239 19257 25 0.8 1223.2 1.0X
+Data column - Json 12387 12393 8 1.3 787.6 1.6X
+Data column - Parquet Vectorized: DataPageV1 227 269 25 69.2 14.5 84.6X
+Data column - Parquet Vectorized: DataPageV2 612 651 28 25.7 38.9 31.4X
+Data column - Parquet MR: DataPageV1 2989 3016 39 5.3 190.0 6.4X
+Data column - Parquet MR: DataPageV2 2750 2754 5 5.7 174.8 7.0X
+Data column - ORC Vectorized 426 467 33 37.0 27.1 45.2X
+Data column - ORC MR 2513 2538 35 6.3 159.8 7.7X
+Partition column - CSV 6623 6627 5 2.4 421.1 2.9X
+Partition column - Json 10234 10235 2 1.5 650.7 1.9X
+Partition column - Parquet Vectorized: DataPageV1 56 73 19 279.8 3.6 342.2X
+Partition column - Parquet Vectorized: DataPageV2 57 72 19 278.3 3.6 340.4X
+Partition column - Parquet MR: DataPageV1 1392 1417 36 11.3 88.5 13.8X
+Partition column - Parquet MR: DataPageV2 1390 1416 37 11.3 88.4 13.8X
+Partition column - ORC Vectorized 56 89 36 283.2 3.5 346.4X
+Partition column - ORC MR 1578 1581 4 10.0 100.4 12.2X
+Both columns - CSV 19178 19181 4 0.8 1219.3 1.0X
+Both columns - Json 13104 13105 1 1.2 833.1 1.5X
+Both columns - Parquet Vectorized: DataPageV1 314 338 21 50.2 19.9 61.4X
+Both columns - Parquet Vectorized: DataPageV2 708 741 54 22.2 45.0 27.2X
+Both columns - Parquet MR: DataPageV1 3083 3105 31 5.1 196.0 6.2X
+Both columns - Parquet MR: DataPageV2 2897 2901 6 5.4 184.2 6.6X
+Both columns - ORC Vectorized 456 504 39 34.5 29.0 42.1X
+Both columns - ORC MR 2594 2597 4 6.1 164.9 7.4X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 9165 9257 130 1.1 874.1 1.0X
+SQL Json 10230 10234 7 1.0 975.6 0.9X
+SQL Parquet Vectorized: DataPageV1 1275 1315 56 8.2 121.6 7.2X
+SQL Parquet Vectorized: DataPageV2 2406 2407 0 4.4 229.5 3.8X
+SQL Parquet MR: DataPageV1 4005 4009 7 2.6 381.9 2.3X
+SQL Parquet MR: DataPageV2 4358 4366 12 2.4 415.6 2.1X
+ParquetReader Vectorized: DataPageV1 985 995 13 10.6 94.0 9.3X
+ParquetReader Vectorized: DataPageV2 2039 2061 32 5.1 194.4 4.5X
+SQL ORC Vectorized 1048 1072 34 10.0 99.9 8.7X
+SQL ORC MR 3179 3196 24 3.3 303.2 2.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 6749 6750 2 1.6 643.6 1.0X
+SQL Json 7960 7967 10 1.3 759.1 0.8X
+SQL Parquet Vectorized: DataPageV1 1078 1105 37 9.7 102.8 6.3X
+SQL Parquet Vectorized: DataPageV2 1939 1941 3 5.4 184.9 3.5X
+SQL Parquet MR: DataPageV1 3090 3099 13 3.4 294.7 2.2X
+SQL Parquet MR: DataPageV2 3274 3286 17 3.2 312.3 2.1X
+ParquetReader Vectorized: DataPageV1 1058 1067 13 9.9 100.9 6.4X
+ParquetReader Vectorized: DataPageV2 1847 1848 2 5.7 176.2 3.7X
+SQL ORC Vectorized 1307 1307 0 8.0 124.6 5.2X
+SQL ORC MR 3078 3122 62 3.4 293.6 2.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 4818 4824 8 2.2 459.5 1.0X
+SQL Json 4853 4878 35 2.2 462.8 1.0X
+SQL Parquet Vectorized: DataPageV1 255 264 6 41.1 24.3 18.9X
+SQL Parquet Vectorized: DataPageV2 711 716 4 14.7 67.8 6.8X
Review comment:
Hmm I wonder why performance dropped so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1018218366
Will this pr speed up string related benchmark result when use Parquet Data Page V2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789320438
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
Review comment:
hmm... may be a stupid question, why does `VectorizedDeltaByteArrayReader` need to hold `valueCount`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r797620913
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
I run the benchmark several times today use GA, It seems that the new pr in `String with Nulls Scan(95%)` scene is still slower than before.
And I run this case locally(use Apple Silicon), there is really no significant difference in M1.
Also cc @dongjoon-hyun @viirya @sunchao for further review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1071140155
@sunchao, @LuciferYang, @sadikovi thank you for your reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r828306457
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,50 +16,127 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader;
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private ByteBuffer previous;
+ private int currentRow = 0;
+
+ // temporary variable used by readBinary
Review comment:
Fixed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -443,6 +444,8 @@ public UTF8String getUTF8String(int rowId) {
}
}
+ public abstract ByteBuffer getBytesUnsafe(int rowId, int count);
Review comment:
I agree as well. @sunchao given that this is from your patch, is it ok to change the name to say `getByteBuffer` ?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java
##########
@@ -221,6 +221,13 @@ protected UTF8String getBytesAsUTF8String(int rowId, int count) {
return UTF8String.fromAddress(null, data + rowId, count);
}
+ @Override
+ public ByteBuffer getBytesUnsafe(int rowId, int count) {
Review comment:
We could, but it would incur an additional function call in performance sensitive code. @sunchao ?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -86,4 +87,20 @@ void readLongsWithRebase(
void write(WritableColumnVector outputColumnVector, int rowId, long val);
}
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
+ void write(WritableColumnVector c, int rowId, ByteBuffer val, int length);
+
+ static void writeArrayByteBuffer(WritableColumnVector c, int rowId, ByteBuffer val,
+ int length) {
+ c.putByteArray(rowId,
+ val.array(),
+ val.arrayOffset() + val.position(),
+ length);
+ }
+
+ static void skipWrite(WritableColumnVector c, int rowId, ByteBuffer val, int length) { }
+
+ }
+
Review comment:
Ok
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader;
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader() {
+ lengthReader = new VectorizedDeltaBinaryPackedReader();
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
+ }
+ currentRow += total;
+ }
+
+ public ByteBuffer getBytes(int rowId) {
+ int length = lengthsVector.getInt(rowId);
+ try {
+ return in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ for (int i = 0; i < total; i++) {
+ int remaining = lengthsVector.getInt(currentRow + i);
+ while (remaining > 0) {
+ remaining -= in.skip(remaining);
+ }
+ }
+ currentRow += total;
+ }
+
Review comment:
OK
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -17,6 +17,7 @@
package org.apache.spark.sql.execution.datasources.parquet;
+import java.nio.ByteBuffer;
Review comment:
OK
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,50 +16,127 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader;
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private ByteBuffer previous;
+ private int currentRow = 0;
+
+ // temporary variable used by readBinary
+ private final WritableColumnVector binaryValVector;
+ // temporary variable used by skipBinary
Review comment:
Fixed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -283,6 +290,11 @@ private void initDataReader(
} catch (IOException e) {
throw new IOException("could not read page in col " + descriptor, e);
}
+ if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding) &&
Review comment:
Added comment. Detailed explanation is in the comment in `VectorizedDeltaByteArrayReader.setPreviousValue`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -367,7 +367,9 @@ private void checkEndOfRowGroup() throws IOException {
datetimeRebaseMode,
datetimeRebaseTz,
int96RebaseMode,
- int96RebaseTz);
+ int96RebaseTz,
+ writerVersion
+ );
Review comment:
OK
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -90,13 +90,18 @@ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOExce
Preconditions.checkArgument(miniSize % 8 == 0,
"miniBlockSize must be multiple of 8, but it's " + miniSize);
this.miniBlockSizeInValues = (int) miniSize;
+ // True value count. May be less than valueCount because of nulls
Review comment:
Added the comment to `getTotalValueCount` as well.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -141,45 +149,54 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
)
val hadoopConf = spark.sessionState.newHadoopConfWithOptions(extraOptions)
- withSQLConf(
- SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",
- ParquetOutputFormat.JOB_SUMMARY_LEVEL -> "ALL") {
- withTempPath { dir =>
- val path = s"${dir.getCanonicalPath}/test.parquet"
-
- val data = (1 to 3).map { i =>
- ( i, i.toLong, i.toShort, Array[Byte](i.toByte), s"test_${i}",
- DateTimeUtils.fromJavaDate(Date.valueOf(s"2021-11-0" + i)),
- DateTimeUtils.fromJavaTimestamp(Timestamp.valueOf(s"2020-11-01 12:00:0" + i)),
- Period.of(1, i, 0), Duration.ofMillis(i * 100),
- new BigDecimal(java.lang.Long.toUnsignedString(i*100000))
- )
+ withMemoryModes { offHeapMode =>
+ withSQLConf(
+ SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> offHeapMode,
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",
+ ParquetOutputFormat.JOB_SUMMARY_LEVEL -> "ALL") {
+ withTempPath { dir =>
+ val path = s"${dir.getCanonicalPath}/test.parquet"
+ // Have more than 2 * 4096 records (so we have multiple tasks and each task
+ // reads at least twice from the reader). This will catch any issues with state
+ // maintained by the reader(s)
+ // Add at least one string with a null
+ val data = (1 to 81971).map { i =>
Review comment:
Yes, it would. Changed. (Sorry that number sneaked in after I tested something else and forgot to undo it).
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaByteArrayEncodingSuite.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.parquet.bytes.DirectByteBufferAllocator
+import org.apache.parquet.column.values.Utils
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayWriter
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{IntegerType, StringType}
+
+/**
+ * Read tests for vectorized Delta byte array reader.
+ * Translated from * org.apache.parquet.column.values.delta.TestDeltaByteArray
+ */
+class ParquetDeltaByteArrayEncodingSuite extends ParquetCompatibilityTest with SharedSparkSession {
+ val values: Array[String] = Array("parquet-mr", "parquet", "parquet-format");
+ val randvalues: Array[String] = Utils.getRandomStringSamples(10000, 32)
+
+ var writer: DeltaByteArrayWriter = _
+ var reader: VectorizedDeltaByteArrayReader = _
+ private var writableColumnVector: WritableColumnVector = _
+
+ protected override def beforeEach(): Unit = {
+ writer = new DeltaByteArrayWriter(64 * 1024, 64 * 1024, new DirectByteBufferAllocator)
+ reader = new VectorizedDeltaByteArrayReader()
+ super.beforeAll()
+ }
+
+ test("test Serialization") {
+ assertReadWrite(writer, reader, values)
+ }
+
+ test("random strings") {
+ assertReadWrite(writer, reader, randvalues)
+ }
+
+ test("random strings with skip") {
+ assertReadWriteWithSkip(writer, reader, randvalues)
+ }
+
+ test("random strings with skipN") {
+ assertReadWriteWithSkipN(writer, reader, randvalues)
+ }
+
+ test("test lengths") {
+ var reader = new VectorizedDeltaBinaryPackedReader
+ Utils.writeData(writer, values)
+ val data = writer.getBytes.toInputStream
+ val length = values.length
+ writableColumnVector = new OnHeapColumnVector(length, IntegerType)
+ reader.initFromPage(length, data)
+ reader.readIntegers(length, writableColumnVector, 0)
+ // test prefix lengths
+ assert(0 == writableColumnVector.getInt(0))
+ assert(7 == writableColumnVector.getInt(1))
+ assert(7 == writableColumnVector.getInt(2))
+
+ reader = new VectorizedDeltaBinaryPackedReader
+ writableColumnVector = new OnHeapColumnVector(length, IntegerType)
+ reader.initFromPage(length, data)
+ reader.readIntegers(length, writableColumnVector, 0)
+ // test suffix lengths
+ assert(10 == writableColumnVector.getInt(0))
+ assert(0 == writableColumnVector.getInt(1))
+ assert(7 == writableColumnVector.getInt(2))
+ }
+
+ private def assertReadWrite(
+ writer: DeltaByteArrayWriter,
+ reader: VectorizedDeltaByteArrayReader,
+ vals: Array[String]): Unit = {
+ Utils.writeData(writer, vals)
+ val length = vals.length
+ val is = writer.getBytes.toInputStream
+
+ writableColumnVector = new OnHeapColumnVector(length, StringType)
+
+ reader.initFromPage(length, is)
+ reader.readBinary(length, writableColumnVector, 0)
+
+ for (i <- 0 until length) {
+ assert(vals(i).getBytes() sameElements writableColumnVector.getBinary(i))
+ }
+ }
+
+ private def assertReadWriteWithSkip(
+ writer: DeltaByteArrayWriter,
+ reader: VectorizedDeltaByteArrayReader,
+ vals: Array[String]): Unit = {
+ Utils.writeData(writer, vals)
+ val length = vals.length
+ val is = writer.getBytes.toInputStream
+ writableColumnVector = new OnHeapColumnVector(length, StringType)
+ reader.initFromPage(length, is)
+ var i = 0
+ while ( {
+ i < vals.length
+ }) {
+ reader.readBinary(1, writableColumnVector, i)
+ assert(vals(i).getBytes() sameElements writableColumnVector.getBinary(i))
+ reader.skipBinary(1)
+ i += 2
+ }
+ }
+
+ private def assertReadWriteWithSkipN(
+ writer: DeltaByteArrayWriter,
+ reader: VectorizedDeltaByteArrayReader,
+ vals: Array[String]): Unit = {
+ Utils.writeData(writer, vals)
+ val length = vals.length
+ val is = writer.getBytes.toInputStream
+ writableColumnVector = new OnHeapColumnVector(length, StringType)
+ reader.initFromPage(length, is)
+ var skipCount = 0
+ var i = 0
+ while ( {
+ i < vals.length
+ }) {
+ skipCount = (vals.length - i) / 2
+ reader.readBinary(1, writableColumnVector, i)
+ assert(vals(i).getBytes() sameElements writableColumnVector.getBinary(i))
+ reader.skipBinary(skipCount)
+ i += skipCount + 1
+ }
+ }
+
Review comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r802234201
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
Review comment:
Ah I see, thanks. @parthchandra can we create a follow-up JIRA and add a TODO here? it'd be better if we respect the config. I think we can make `VectorizedColumnReader` a `Closable` in future and close it `VectorizedParquetRecordReader.close()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r806364300
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r806364300
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r810289632
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 14859 14914 77 1.1 944.7 1.0X
+SQL Json 9418 9457 55 1.7 598.8 1.6X
+SQL Parquet Vectorized: DataPageV1 109 128 14 144.6 6.9 136.6X
+SQL Parquet Vectorized: DataPageV2 79 89 8 199.3 5.0 188.3X
+SQL Parquet MR: DataPageV1 1699 1743 62 9.3 108.0 8.7X
+SQL Parquet MR: DataPageV2 1462 1489 38 10.8 93.0 10.2X
+SQL ORC Vectorized 165 200 33 95.3 10.5 90.0X
+SQL ORC MR 1409 1420 16 11.2 89.6 10.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 101 104 3 155.2 6.4 1.0X
+ParquetReader Vectorized: DataPageV2 82 85 5 192.0 5.2 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 48 50 2 324.6 3.1 2.1X
+ParquetReader Vectorized -> Row: DataPageV2 29 31 3 539.4 1.9 3.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 17619 17639 28 0.9 1120.2 1.0X
+SQL Json 10590 10606 23 1.5 673.3 1.7X
+SQL Parquet Vectorized: DataPageV1 178 194 10 88.2 11.3 98.8X
+SQL Parquet Vectorized: DataPageV2 178 188 9 88.2 11.3 98.7X
+SQL Parquet MR: DataPageV1 1884 1887 4 8.4 119.8 9.4X
+SQL Parquet MR: DataPageV2 1689 1742 75 9.3 107.4 10.4X
+SQL ORC Vectorized 162 193 24 97.0 10.3 108.7X
+SQL ORC MR 1505 1552 67 10.5 95.7 11.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 230 236 13 68.3 14.6 1.0X
+ParquetReader Vectorized: DataPageV2 228 233 8 69.1 14.5 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 138 150 26 113.7 8.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 137 140 2 114.5 8.7 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 18488 18494 8 0.9 1175.5 1.0X
+SQL Json 11190 11195 7 1.4 711.4 1.7X
+SQL Parquet Vectorized: DataPageV1 125 155 34 125.7 8.0 147.7X
+SQL Parquet Vectorized: DataPageV2 183 192 9 86.1 11.6 101.2X
+SQL Parquet MR: DataPageV1 2153 2160 10 7.3 136.9 8.6X
+SQL Parquet MR: DataPageV2 1876 1889 18 8.4 119.3 9.9X
+SQL ORC Vectorized 212 257 23 74.4 13.4 87.4X
+SQL ORC MR 1653 1658 7 9.5 105.1 11.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 198 201 5 79.6 12.6 1.0X
+ParquetReader Vectorized: DataPageV2 256 260 3 61.5 16.3 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 193 226 14 81.4 12.3 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 250 253 2 62.8 15.9 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 18614 18703 125 0.8 1183.5 1.0X
+SQL Json 11673 11711 53 1.3 742.2 1.6X
+SQL Parquet Vectorized: DataPageV1 128 154 26 123.1 8.1 145.7X
+SQL Parquet Vectorized: DataPageV2 270 302 23 58.3 17.1 69.0X
+SQL Parquet MR: DataPageV1 2117 2145 39 7.4 134.6 8.8X
+SQL Parquet MR: DataPageV2 1855 1860 7 8.5 117.9 10.0X
+SQL ORC Vectorized 277 292 16 56.7 17.6 67.2X
+SQL ORC MR 1623 1629 9 9.7 103.2 11.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 225 226 1 69.9 14.3 1.0X
+ParquetReader Vectorized: DataPageV2 362 365 2 43.4 23.0 0.6X
+ParquetReader Vectorized -> Row: DataPageV1 193 218 18 81.5 12.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV2 360 366 6 43.7 22.9 0.6X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 23939 23953 20 0.7 1522.0 1.0X
+SQL Json 14445 14449 5 1.1 918.4 1.7X
+SQL Parquet Vectorized: DataPageV1 186 229 28 84.7 11.8 128.9X
+SQL Parquet Vectorized: DataPageV2 459 493 25 34.3 29.2 52.2X
+SQL Parquet MR: DataPageV1 2180 2184 7 7.2 138.6 11.0X
+SQL Parquet MR: DataPageV2 1954 1973 27 8.1 124.2 12.3X
+SQL ORC Vectorized 368 392 24 42.8 23.4 65.1X
+SQL ORC MR 1793 1794 2 8.8 114.0 13.4X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 280 293 18 56.1 17.8 1.0X
+ParquetReader Vectorized: DataPageV2 577 602 48 27.3 36.7 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 314 321 10 50.1 19.9 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 581 584 4 27.1 37.0 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 19278 19291 18 0.8 1225.6 1.0X
+SQL Json 13366 13381 21 1.2 849.8 1.4X
+SQL Parquet Vectorized: DataPageV1 130 152 23 120.8 8.3 148.1X
+SQL Parquet Vectorized: DataPageV2 135 157 17 116.8 8.6 143.2X
+SQL Parquet MR: DataPageV1 2126 2137 15 7.4 135.2 9.1X
+SQL Parquet MR: DataPageV2 1970 1985 21 8.0 125.2 9.8X
+SQL ORC Vectorized 387 396 11 40.7 24.6 49.8X
+SQL ORC MR 1831 1832 1 8.6 116.4 10.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 194 197 5 81.1 12.3 1.0X
+ParquetReader Vectorized: DataPageV2 194 197 7 81.2 12.3 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 225 253 18 69.9 14.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 224 252 18 70.2 14.2 0.9X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 24691 24705 19 0.6 1569.8 1.0X
+SQL Json 18028 18028 0 0.9 1146.2 1.4X
+SQL Parquet Vectorized: DataPageV1 190 225 28 83.0 12.0 130.3X
+SQL Parquet Vectorized: DataPageV2 188 230 26 83.9 11.9 131.7X
+SQL Parquet MR: DataPageV1 2362 2365 4 6.7 150.2 10.5X
+SQL Parquet MR: DataPageV2 2061 2078 25 7.6 131.0 12.0X
+SQL ORC Vectorized 499 524 37 31.6 31.7 49.5X
+SQL ORC MR 1870 1880 14 8.4 118.9 13.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 276 295 21 57.0 17.5 1.0X
+ParquetReader Vectorized: DataPageV2 278 289 17 56.6 17.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 315 326 15 50.0 20.0 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 315 323 8 49.9 20.0 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 16840 16908 96 0.6 1606.0 1.0X
+SQL Json 12496 12513 25 0.8 1191.7 1.3X
+SQL Parquet Vectorized: DataPageV1 2169 2172 5 4.8 206.9 7.8X
+SQL Parquet Vectorized: DataPageV2 3102 3119 24 3.4 295.9 5.4X
+SQL Parquet MR: DataPageV1 4140 4144 5 2.5 394.8 4.1X
+SQL Parquet MR: DataPageV2 3988 3996 12 2.6 380.3 4.2X
+SQL ORC Vectorized 2180 2196 23 4.8 207.9 7.7X
+SQL ORC MR 3765 3766 2 2.8 359.0 4.5X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 9960 9960 0 1.1 949.8 1.0X
+SQL Json 7625 7712 123 1.4 727.2 1.3X
+SQL Parquet Vectorized: DataPageV1 577 582 6 18.2 55.0 17.3X
+SQL Parquet Vectorized: DataPageV2 584 592 6 18.0 55.7 17.1X
+SQL Parquet MR: DataPageV1 1722 1736 19 6.1 164.2 5.8X
+SQL Parquet MR: DataPageV2 1662 1668 9 6.3 158.5 6.0X
+SQL ORC Vectorized 483 524 27 21.7 46.1 20.6X
+SQL ORC MR 1841 1850 14 5.7 175.5 5.4X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 23787 23788 2 0.7 1512.3 1.0X
+Data column - Json 13993 14011 25 1.1 889.7 1.7X
+Data column - Parquet Vectorized: DataPageV1 184 235 36 85.4 11.7 129.2X
+Data column - Parquet Vectorized: DataPageV2 531 542 15 29.6 33.7 44.8X
+Data column - Parquet MR: DataPageV1 2539 2547 13 6.2 161.4 9.4X
+Data column - Parquet MR: DataPageV2 2299 2301 3 6.8 146.2 10.3X
+Data column - ORC Vectorized 379 403 23 41.5 24.1 62.8X
+Data column - ORC MR 2047 2070 33 7.7 130.1 11.6X
+Partition column - CSV 6834 6835 1 2.3 434.5 3.5X
+Partition column - Json 11444 11478 49 1.4 727.6 2.1X
+Partition column - Parquet Vectorized: DataPageV1 51 71 22 308.6 3.2 466.7X
+Partition column - Parquet Vectorized: DataPageV2 51 61 16 310.5 3.2 469.5X
+Partition column - Parquet MR: DataPageV1 1203 1214 15 13.1 76.5 19.8X
+Partition column - Parquet MR: DataPageV2 1210 1224 20 13.0 76.9 19.7X
+Partition column - ORC Vectorized 52 67 14 303.1 3.3 458.4X
+Partition column - ORC MR 1338 1342 5 11.8 85.1 17.8X
+Both columns - CSV 24051 24052 2 0.7 1529.1 1.0X
+Both columns - Json 15016 15030 20 1.0 954.7 1.6X
+Both columns - Parquet Vectorized: DataPageV1 235 269 27 66.9 15.0 101.2X
+Both columns - Parquet Vectorized: DataPageV2 563 617 60 27.9 35.8 42.2X
+Both columns - Parquet MR: DataPageV1 2525 2555 43 6.2 160.5 9.4X
+Both columns - Parquet MR: DataPageV2 2256 2267 15 7.0 143.5 10.5X
+Both columns - ORC Vectorized 407 454 51 38.7 25.9 58.5X
+Both columns - ORC MR 2153 2155 2 7.3 136.9 11.0X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 11812 11849 53 0.9 1126.4 1.0X
+SQL Json 11454 11467 18 0.9 1092.3 1.0X
+SQL Parquet Vectorized: DataPageV1 1250 1276 37 8.4 119.2 9.5X
+SQL Parquet Vectorized: DataPageV2 2248 2261 17 4.7 214.4 5.3X
+SQL Parquet MR: DataPageV1 3629 3630 1 2.9 346.1 3.3X
+SQL Parquet MR: DataPageV2 3929 3934 6 2.7 374.7 3.0X
+ParquetReader Vectorized: DataPageV1 921 922 2 11.4 87.8 12.8X
+ParquetReader Vectorized: DataPageV2 1890 1890 0 5.5 180.3 6.2X
+SQL ORC Vectorized 1079 1105 36 9.7 102.9 10.9X
+SQL ORC MR 3042 3070 40 3.4 290.1 3.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 8399 8410 16 1.2 801.0 1.0X
+SQL Json 8892 8905 18 1.2 848.0 0.9X
+SQL Parquet Vectorized: DataPageV1 1065 1092 38 9.8 101.6 7.9X
+SQL Parquet Vectorized: DataPageV2 1747 1747 0 6.0 166.6 4.8X
+SQL Parquet MR: DataPageV1 2718 2719 1 3.9 259.2 3.1X
+SQL Parquet MR: DataPageV2 2955 2964 12 3.5 281.8 2.8X
+ParquetReader Vectorized: DataPageV1 1082 1084 3 9.7 103.2 7.8X
+ParquetReader Vectorized: DataPageV2 1707 1713 9 6.1 162.8 4.9X
+SQL ORC Vectorized 1345 1357 17 7.8 128.3 6.2X
+SQL ORC MR 3012 3046 47 3.5 287.3 2.8X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 6169 6176 10 1.7 588.3 1.0X
+SQL Json 5352 5376 35 2.0 510.4 1.2X
+SQL Parquet Vectorized: DataPageV1 248 255 7 42.3 23.6 24.9X
+SQL Parquet Vectorized: DataPageV2 364 372 10 28.8 34.7 17.0X
+SQL Parquet MR: DataPageV1 1624 1626 3 6.5 154.9 3.8X
+SQL Parquet MR: DataPageV2 1520 1526 8 6.9 145.0 4.1X
+ParquetReader Vectorized: DataPageV1 259 262 1 40.4 24.7 23.8X
+ParquetReader Vectorized: DataPageV2 376 378 2 27.9 35.9 16.4X
Review comment:
I was looking at numbers of "ParquetReader Vectorized: DataPageV2" vs "ParquetReader Vectorized: DataPageV1". Previously it is 15.6/19.5 = 0.8, while the new ratio is 16.4/23.8 = 0.69. The other two cases (0% and 50% nulls) look OK though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #35262: [SPARK-36879][SQL][FOLLOWUP] Support Parquet v2 data page encodings for the vectorized path
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1017818566
Will do, @parthchandra could you use a different JIRA for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1018218366
Will this pr speed up string related benchmark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r792232599
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,6 +96,12 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
HadoopInputFile.fromPath(file, configuration), options);
this.reader = new ParquetRowGroupReaderImpl(fileReader);
this.fileSchema = fileReader.getFileMetaData().getSchema();
+ try {
+ this.writerVersion = VersionParser.parse(fileReader.getFileMetaData().getCreatedBy());
+ } catch (Exception e) {
Review comment:
Well yes. I encountered at least one case where the version information was empty and the version check threw a NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794236177
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
@parthchandra if `This method gets called only for non-null values.`, then there should be no need to add `lengthsVector.putInts(0, valueCount, 0)` because the place where the exception was thrown was originally a null value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794238824
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
It seems that there are many strings with length of zero when `String with Nulls Scan` is running. I'm not sure whether there is a problem with the design of `String with Nulls Scan`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1021817183
> Updated the JDK 8 benchmark results as well.
After comparing the new bench data, I find that the data corresponding to `Parquet Data Page V2` in the two test cases `String with Nulls Scan (50.0%) ` and `String with Nulls Scan (95.0%)` is relatively slower than the previous pr (although the CPU frequency of the testing machine is reduced):
| | before | after |
| :--- | :--- |:--- |
| String with Nulls Scan (50.0%) | 145.7 ns/per row | 228.7 ns/per row |
| String with Nulls Scan (95.0%) | 25.2 ns/per row | 77.9 ns/per row |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1070086327
Thanks @sadikovi for the review! looks like we need to fix lint. Since Spark 3.3 branch has been cut, I've asked in the dev mailing list to see if we can still include this in the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r790434627
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ prefixLengthVector = new OffHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OffHeapColumnVector(valueCount, BinaryType);
+ } else {
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ }
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, null, 0,
+ (w, r, v, l) ->
Review comment:
I changed `parquet v2 pages - delta encoding` in `ParquetEncodingSuite` into a circular query
```
while (true) {
val actual = spark.read.parquet(path).collect()
assert(actual.sortBy(_.getInt(0)) === data.map(Row.fromTuple))
}
```
and dumped the memory many times.
Then I found there are many object of class
`org.apache.spark.sql.execution.datasources.parquet.VectorizedDeltaByteArrayReader$$Lambda$3232` in memory dump:
It seems that because the lambda involves an external variable `binaryVal`, a new object will be generated every time when the method called @parthchandra ,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1029288528
@LuciferYang you had a comment that seems to have vanished. From the email -
> In [sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java](https://github.com/apache/spark/pull/35262#discussion_r797417782):
>
> > }
> + outputWriter.write(c, rowId + i, ByteBuffer.wrap(previous), previous.length);
> Is line106 always equivalent to c.putByteArray(rowId, previous, 0, length)?
There are multiple output writers. The method called in `readBinary` is equivalent to `c.putByteArray(rowId, previous, 0, length)`; however, if (and when) we add back the OFF_HEAP option, this method will change depending on the memory mode.
Also `skipBytes` also calls the same `readValues` method but with a different output writer which does nothing. Note that we still have to process each value even if we do not write it out (because we need the previous value to compute every subsequent value).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794253952
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Or maybe`lengthsVector ` can't distinguish between zero-length string and Null in `readBinary`?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Or maybe`lengthsVector ` can't distinguish between zero-length string and Null in `readBinary` scope?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794253952
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Or `lengthsVector ` can't distinguish between zero-length string and Null?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Or maybe`lengthsVector ` can't distinguish between zero-length string and Null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1024049100
I have another question:
When `memoryMode` is `MemoryMode.OFF_HEAP`, some `OffHeapColumnVector` will be create like `binaryValVector`, `lengthsVector` and so on. `OffHeapColumnVector` needs to call the `close` method manually to release the allocated memory, but I can't find the corresponding call.
SO there may be a memory leak when `memoryMode` is `MemoryMode.OFF_HEAP`. @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1022919561
@parthchandra I think we should add some UTs similar to `String with Nulls Scan` because when I add
```
sparkSession.conf.set(SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key, "true")
```
to `DataSourceReadBenchmark` to enable `ColumnVector` use offheap memory, `String with Nulls Scan` releated cases will failed as follows:
```
14:33:29.271 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 5043.0 (TID 3936)
org.apache.spark.sql.execution.QueryExecutionException: Encountered error while reading file file:///private/var/folders/0x/xj61_dbd0dldn793s6cyb7rr0000gp/T/spark-a6065795-c141-43cd-8ec6-359f3f3a0307/parquetV2/part-00000-7c6de322-95b1-4283-9399-8306753c68ab-c000.snappy.parquet. Details:
at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:659) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:116) ~[classes/:?]
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:546) ~[classes/:?]
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source) ~[?:?]
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source) ~[?:?]
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) ~[?:?]
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) ~[classes/:?]
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760) ~[classes/:?]
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460) ~[scala-library-2.12.15.jar:?]
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140) ~[classes/:?]
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) ~[classes/:?]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) ~[classes/:?]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) ~[classes/:?]
at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[classes/:?]
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:507) ~[classes/:?]
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1475) ~[classes/:?]
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:510) [classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read 268435456 bytes
at org.apache.spark.sql.execution.datasources.parquet.VectorizedDeltaLengthByteArrayReader.readBinary(VectorizedDeltaLengthByteArrayReader.java:79) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedDeltaByteArrayReader.initFromPage(VectorizedDeltaByteArrayReader.java:76) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.initDataReader(VectorizedColumnReader.java:293) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV2(VectorizedColumnReader.java:362) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.access$100(VectorizedColumnReader.java:52) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:260) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:247) ~[classes/:?]
at org.apache.parquet.column.page.DataPageV2.accept(DataPageV2.java:192) ~[parquet-column-1.12.2.jar:1.12.2]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPage(VectorizedColumnReader.java:247) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:183) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:311) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:209) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:116) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:274) ~[classes/:?]
... 19 more
```
I manually verified that there was no such problem before this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789351038
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -86,4 +87,26 @@ void readLongsWithRebase(
void write(WritableColumnVector outputColumnVector, int rowId, long val);
}
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
+ void write(WritableColumnVector c, int rowId, ByteBuffer val, int length);
+
+ static void writeArrayByteBuffer(WritableColumnVector c, int rowId, ByteBuffer val,
Review comment:
Is it a good practice to add `static` methods to interface? I'm not sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1085065101
Thank you @sunchao , @LuciferYang , @sadikovi . I'll submit a few smaller, followup PRs for the issues that were deferred as a result of the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r826764705
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader;
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader() {
+ lengthReader = new VectorizedDeltaBinaryPackedReader();
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
+ }
+ currentRow += total;
+ }
+
+ public ByteBuffer getBytes(int rowId) {
+ int length = lengthsVector.getInt(rowId);
+ try {
+ return in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(currentRow + i);
+ int remaining = length;
+ while (remaining > 0) {
+ remaining -= in.skip(remaining);
Review comment:
Did I miss anything? Do we really need `length` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r820013541
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,115 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader;
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ // temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.prefixLengthReader = new VectorizedDeltaBinaryPackedReader();
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(prefixLengthReader.getTotalValueCount(), suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ // NOTE: due to PARQUET-246, it is important that we
+ // respect prefixLength which was read from prefixLengthReader,
+ // even for the *first* value of a page. Even though the first
+ // value of the page should have an empty prefix, it may not
+ // because of PARQUET-246.
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ int length = prefixLength + suffix.length;
+
+ // We have to do this to materialize the output
+ if (prefixLength != 0) {
+ // We could do
+ // c.putByteArray(rowId + i, previous, 0, prefixLength);
+ // c.putByteArray(rowId+i, suffix, prefixLength, suffix.length);
+ // previous = c.getBinary(rowId+1);
+ // but it incurs the same cost of copying the values twice _and_ c.getBinary
+ // is a _slow_ byte by byte copy
+ // The following always uses the faster system arraycopy method
+ byte[] out = new byte[length];
Review comment:
We can also potentially skip this copying at least for `OnHeapColumnVector`. [I tried it](https://github.com/sunchao/spark/commit/95f272c39a19cb1abdbc6c79e96b8fae435a96ab) and it gives some extra performance improvements.
```
[info] OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Mac OS X 10.16
[info] Intel(R) Core(TM) i9-10910 CPU @ 3.60GHz
[info] String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] SQL CSV 5721 5727 8 1.8 545.6 1.0X
[info] SQL Json 6289 6295 9 1.7 599.7 0.9X
[info] SQL Parquet Vectorized: DataPageV1 700 800 87 15.0 66.7 8.2X
[info] SQL Parquet Vectorized: DataPageV2 994 1031 52 10.5 94.8 5.8X
[info] SQL Parquet MR: DataPageV1 2035 2051 23 5.2 194.1 2.8X
[info] SQL Parquet MR: DataPageV2 2289 2454 232 4.6 218.3 2.5X
[info] ParquetReader Vectorized: DataPageV1 472 482 15 22.2 45.0 12.1X
[info] ParquetReader Vectorized: DataPageV2 640 645 4 16.4 61.0 8.9X
[info] SQL ORC Vectorized 670 694 35 15.7 63.9 8.5X
[info] SQL ORC MR 1846 2047 284 5.7 176.0 3.1X
[info] OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Mac OS X 10.16
[info] Intel(R) Core(TM) i9-10910 CPU @ 3.60GHz
[info] String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] SQL CSV 4825 4890 91 2.2 460.2 1.0X
[info] SQL Json 5298 7385 2951 2.0 505.3 0.9X
[info] SQL Parquet Vectorized: DataPageV1 701 889 169 14.9 66.9 6.9X
[info] SQL Parquet Vectorized: DataPageV2 684 737 58 15.3 65.2 7.1X
[info] SQL Parquet MR: DataPageV1 1857 1869 17 5.6 177.1 2.6X
[info] SQL Parquet MR: DataPageV2 2034 2146 159 5.2 193.9 2.4X
[info] ParquetReader Vectorized: DataPageV1 474 493 11 22.1 45.2 10.2X
[info] ParquetReader Vectorized: DataPageV2 585 586 1 17.9 55.8 8.2X
[info] SQL ORC Vectorized 810 845 53 12.9 77.3 6.0X
[info] SQL ORC MR 1854 1935 114 5.7 176.8 2.6X
[info] OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Mac OS X 10.16
[info] Intel(R) Core(TM) i9-10910 CPU @ 3.60GHz
[info] String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] SQL CSV 3212 3256 63 3.3 306.3 1.0X
[info] SQL Json 3693 3695 3 2.8 352.2 0.9X
[info] SQL Parquet Vectorized: DataPageV1 147 203 46 71.2 14.0 21.8X
[info] SQL Parquet Vectorized: DataPageV2 160 286 144 65.4 15.3 20.0X
[info] SQL Parquet MR: DataPageV1 1229 1351 172 8.5 117.2 2.6X
[info] SQL Parquet MR: DataPageV2 1074 1099 36 9.8 102.4 3.0X
[info] ParquetReader Vectorized: DataPageV1 107 109 2 97.9 10.2 30.0X
[info] ParquetReader Vectorized: DataPageV2 124 127 2 84.7 11.8 25.9X
[info] SQL ORC Vectorized 262 308 86 40.0 25.0 12.3X
[info] SQL ORC MR 1002 1070 96 10.5 95.5 3.2X
``
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1017861246
Created a new JIRA and modified the commit message(s) and the PR title.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789342727
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ prefixLengthVector = new OffHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OffHeapColumnVector(valueCount, BinaryType);
+ } else {
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ }
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, null, 0,
+ (w, r, v, l) ->
Review comment:
Does this lambda will frequent object creation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r810277628
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 14859 14914 77 1.1 944.7 1.0X
+SQL Json 9418 9457 55 1.7 598.8 1.6X
+SQL Parquet Vectorized: DataPageV1 109 128 14 144.6 6.9 136.6X
+SQL Parquet Vectorized: DataPageV2 79 89 8 199.3 5.0 188.3X
+SQL Parquet MR: DataPageV1 1699 1743 62 9.3 108.0 8.7X
+SQL Parquet MR: DataPageV2 1462 1489 38 10.8 93.0 10.2X
+SQL ORC Vectorized 165 200 33 95.3 10.5 90.0X
+SQL ORC MR 1409 1420 16 11.2 89.6 10.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 101 104 3 155.2 6.4 1.0X
+ParquetReader Vectorized: DataPageV2 82 85 5 192.0 5.2 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 48 50 2 324.6 3.1 2.1X
+ParquetReader Vectorized -> Row: DataPageV2 29 31 3 539.4 1.9 3.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 17619 17639 28 0.9 1120.2 1.0X
+SQL Json 10590 10606 23 1.5 673.3 1.7X
+SQL Parquet Vectorized: DataPageV1 178 194 10 88.2 11.3 98.8X
+SQL Parquet Vectorized: DataPageV2 178 188 9 88.2 11.3 98.7X
+SQL Parquet MR: DataPageV1 1884 1887 4 8.4 119.8 9.4X
+SQL Parquet MR: DataPageV2 1689 1742 75 9.3 107.4 10.4X
+SQL ORC Vectorized 162 193 24 97.0 10.3 108.7X
+SQL ORC MR 1505 1552 67 10.5 95.7 11.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 230 236 13 68.3 14.6 1.0X
+ParquetReader Vectorized: DataPageV2 228 233 8 69.1 14.5 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 138 150 26 113.7 8.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 137 140 2 114.5 8.7 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 18488 18494 8 0.9 1175.5 1.0X
+SQL Json 11190 11195 7 1.4 711.4 1.7X
+SQL Parquet Vectorized: DataPageV1 125 155 34 125.7 8.0 147.7X
+SQL Parquet Vectorized: DataPageV2 183 192 9 86.1 11.6 101.2X
+SQL Parquet MR: DataPageV1 2153 2160 10 7.3 136.9 8.6X
+SQL Parquet MR: DataPageV2 1876 1889 18 8.4 119.3 9.9X
+SQL ORC Vectorized 212 257 23 74.4 13.4 87.4X
+SQL ORC MR 1653 1658 7 9.5 105.1 11.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 198 201 5 79.6 12.6 1.0X
+ParquetReader Vectorized: DataPageV2 256 260 3 61.5 16.3 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 193 226 14 81.4 12.3 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 250 253 2 62.8 15.9 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 18614 18703 125 0.8 1183.5 1.0X
+SQL Json 11673 11711 53 1.3 742.2 1.6X
+SQL Parquet Vectorized: DataPageV1 128 154 26 123.1 8.1 145.7X
+SQL Parquet Vectorized: DataPageV2 270 302 23 58.3 17.1 69.0X
+SQL Parquet MR: DataPageV1 2117 2145 39 7.4 134.6 8.8X
+SQL Parquet MR: DataPageV2 1855 1860 7 8.5 117.9 10.0X
+SQL ORC Vectorized 277 292 16 56.7 17.6 67.2X
+SQL ORC MR 1623 1629 9 9.7 103.2 11.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 225 226 1 69.9 14.3 1.0X
+ParquetReader Vectorized: DataPageV2 362 365 2 43.4 23.0 0.6X
+ParquetReader Vectorized -> Row: DataPageV1 193 218 18 81.5 12.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV2 360 366 6 43.7 22.9 0.6X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 23939 23953 20 0.7 1522.0 1.0X
+SQL Json 14445 14449 5 1.1 918.4 1.7X
+SQL Parquet Vectorized: DataPageV1 186 229 28 84.7 11.8 128.9X
+SQL Parquet Vectorized: DataPageV2 459 493 25 34.3 29.2 52.2X
+SQL Parquet MR: DataPageV1 2180 2184 7 7.2 138.6 11.0X
+SQL Parquet MR: DataPageV2 1954 1973 27 8.1 124.2 12.3X
+SQL ORC Vectorized 368 392 24 42.8 23.4 65.1X
+SQL ORC MR 1793 1794 2 8.8 114.0 13.4X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 280 293 18 56.1 17.8 1.0X
+ParquetReader Vectorized: DataPageV2 577 602 48 27.3 36.7 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 314 321 10 50.1 19.9 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 581 584 4 27.1 37.0 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 19278 19291 18 0.8 1225.6 1.0X
+SQL Json 13366 13381 21 1.2 849.8 1.4X
+SQL Parquet Vectorized: DataPageV1 130 152 23 120.8 8.3 148.1X
+SQL Parquet Vectorized: DataPageV2 135 157 17 116.8 8.6 143.2X
+SQL Parquet MR: DataPageV1 2126 2137 15 7.4 135.2 9.1X
+SQL Parquet MR: DataPageV2 1970 1985 21 8.0 125.2 9.8X
+SQL ORC Vectorized 387 396 11 40.7 24.6 49.8X
+SQL ORC MR 1831 1832 1 8.6 116.4 10.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 194 197 5 81.1 12.3 1.0X
+ParquetReader Vectorized: DataPageV2 194 197 7 81.2 12.3 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 225 253 18 69.9 14.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 224 252 18 70.2 14.2 0.9X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 24691 24705 19 0.6 1569.8 1.0X
+SQL Json 18028 18028 0 0.9 1146.2 1.4X
+SQL Parquet Vectorized: DataPageV1 190 225 28 83.0 12.0 130.3X
+SQL Parquet Vectorized: DataPageV2 188 230 26 83.9 11.9 131.7X
+SQL Parquet MR: DataPageV1 2362 2365 4 6.7 150.2 10.5X
+SQL Parquet MR: DataPageV2 2061 2078 25 7.6 131.0 12.0X
+SQL ORC Vectorized 499 524 37 31.6 31.7 49.5X
+SQL ORC MR 1870 1880 14 8.4 118.9 13.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 276 295 21 57.0 17.5 1.0X
+ParquetReader Vectorized: DataPageV2 278 289 17 56.6 17.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 315 326 15 50.0 20.0 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 315 323 8 49.9 20.0 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 16840 16908 96 0.6 1606.0 1.0X
+SQL Json 12496 12513 25 0.8 1191.7 1.3X
+SQL Parquet Vectorized: DataPageV1 2169 2172 5 4.8 206.9 7.8X
+SQL Parquet Vectorized: DataPageV2 3102 3119 24 3.4 295.9 5.4X
+SQL Parquet MR: DataPageV1 4140 4144 5 2.5 394.8 4.1X
+SQL Parquet MR: DataPageV2 3988 3996 12 2.6 380.3 4.2X
+SQL ORC Vectorized 2180 2196 23 4.8 207.9 7.7X
+SQL ORC MR 3765 3766 2 2.8 359.0 4.5X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 9960 9960 0 1.1 949.8 1.0X
+SQL Json 7625 7712 123 1.4 727.2 1.3X
+SQL Parquet Vectorized: DataPageV1 577 582 6 18.2 55.0 17.3X
+SQL Parquet Vectorized: DataPageV2 584 592 6 18.0 55.7 17.1X
+SQL Parquet MR: DataPageV1 1722 1736 19 6.1 164.2 5.8X
+SQL Parquet MR: DataPageV2 1662 1668 9 6.3 158.5 6.0X
+SQL ORC Vectorized 483 524 27 21.7 46.1 20.6X
+SQL ORC MR 1841 1850 14 5.7 175.5 5.4X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 23787 23788 2 0.7 1512.3 1.0X
+Data column - Json 13993 14011 25 1.1 889.7 1.7X
+Data column - Parquet Vectorized: DataPageV1 184 235 36 85.4 11.7 129.2X
+Data column - Parquet Vectorized: DataPageV2 531 542 15 29.6 33.7 44.8X
+Data column - Parquet MR: DataPageV1 2539 2547 13 6.2 161.4 9.4X
+Data column - Parquet MR: DataPageV2 2299 2301 3 6.8 146.2 10.3X
+Data column - ORC Vectorized 379 403 23 41.5 24.1 62.8X
+Data column - ORC MR 2047 2070 33 7.7 130.1 11.6X
+Partition column - CSV 6834 6835 1 2.3 434.5 3.5X
+Partition column - Json 11444 11478 49 1.4 727.6 2.1X
+Partition column - Parquet Vectorized: DataPageV1 51 71 22 308.6 3.2 466.7X
+Partition column - Parquet Vectorized: DataPageV2 51 61 16 310.5 3.2 469.5X
+Partition column - Parquet MR: DataPageV1 1203 1214 15 13.1 76.5 19.8X
+Partition column - Parquet MR: DataPageV2 1210 1224 20 13.0 76.9 19.7X
+Partition column - ORC Vectorized 52 67 14 303.1 3.3 458.4X
+Partition column - ORC MR 1338 1342 5 11.8 85.1 17.8X
+Both columns - CSV 24051 24052 2 0.7 1529.1 1.0X
+Both columns - Json 15016 15030 20 1.0 954.7 1.6X
+Both columns - Parquet Vectorized: DataPageV1 235 269 27 66.9 15.0 101.2X
+Both columns - Parquet Vectorized: DataPageV2 563 617 60 27.9 35.8 42.2X
+Both columns - Parquet MR: DataPageV1 2525 2555 43 6.2 160.5 9.4X
+Both columns - Parquet MR: DataPageV2 2256 2267 15 7.0 143.5 10.5X
+Both columns - ORC Vectorized 407 454 51 38.7 25.9 58.5X
+Both columns - ORC MR 2153 2155 2 7.3 136.9 11.0X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 11812 11849 53 0.9 1126.4 1.0X
+SQL Json 11454 11467 18 0.9 1092.3 1.0X
+SQL Parquet Vectorized: DataPageV1 1250 1276 37 8.4 119.2 9.5X
+SQL Parquet Vectorized: DataPageV2 2248 2261 17 4.7 214.4 5.3X
+SQL Parquet MR: DataPageV1 3629 3630 1 2.9 346.1 3.3X
+SQL Parquet MR: DataPageV2 3929 3934 6 2.7 374.7 3.0X
+ParquetReader Vectorized: DataPageV1 921 922 2 11.4 87.8 12.8X
+ParquetReader Vectorized: DataPageV2 1890 1890 0 5.5 180.3 6.2X
+SQL ORC Vectorized 1079 1105 36 9.7 102.9 10.9X
+SQL ORC MR 3042 3070 40 3.4 290.1 3.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 8399 8410 16 1.2 801.0 1.0X
+SQL Json 8892 8905 18 1.2 848.0 0.9X
+SQL Parquet Vectorized: DataPageV1 1065 1092 38 9.8 101.6 7.9X
+SQL Parquet Vectorized: DataPageV2 1747 1747 0 6.0 166.6 4.8X
+SQL Parquet MR: DataPageV1 2718 2719 1 3.9 259.2 3.1X
+SQL Parquet MR: DataPageV2 2955 2964 12 3.5 281.8 2.8X
+ParquetReader Vectorized: DataPageV1 1082 1084 3 9.7 103.2 7.8X
+ParquetReader Vectorized: DataPageV2 1707 1713 9 6.1 162.8 4.9X
+SQL ORC Vectorized 1345 1357 17 7.8 128.3 6.2X
+SQL ORC MR 3012 3046 47 3.5 287.3 2.8X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 6169 6176 10 1.7 588.3 1.0X
+SQL Json 5352 5376 35 2.0 510.4 1.2X
+SQL Parquet Vectorized: DataPageV1 248 255 7 42.3 23.6 24.9X
+SQL Parquet Vectorized: DataPageV2 364 372 10 28.8 34.7 17.0X
+SQL Parquet MR: DataPageV1 1624 1626 3 6.5 154.9 3.8X
+SQL Parquet MR: DataPageV2 1520 1526 8 6.9 145.0 4.1X
+ParquetReader Vectorized: DataPageV1 259 262 1 40.4 24.7 23.8X
+ParquetReader Vectorized: DataPageV2 376 378 2 27.9 35.9 16.4X
Review comment:
Which line(s) are you looking at? Not sure which numbers are 69%, and 80% that you're referring to.
One reason vectorization isn't helping (maybe) is because the calling code is calling the vectorized reader one row at at time.
(https://github.com/apache/spark/blob/439975590cf4f21c2a548a2ac6231eb234e1a2f3/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java#L239)
Let me address this in another PR. (https://issues.apache.org/jira/browse/SPARK-38254)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1024049100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794168786
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
if `length` is 0, can we simplify the operation of `line77 ~ line82`
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
so if `length` is 0, can we simplify the operation of `line77 ~ line82`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r802222581
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
Review comment:
As discussed earlier (https://github.com/apache/spark/pull/35262#issuecomment-1025369374), if we use `OffHeapColumnVector `, we need to consider the memory leak issue because there is no path to call `OffHeapColumnVector.close()` method currently, this will make the current pr more complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1025369374
> Oh dear. This will take some fixing. Thank you for catching this!
I think we can also remove the `OFF_HEAP` mode support in current pr, (Sorry, I haven't seen the advantages of `OFF_HEAP` mode in the current scenario and other readers haven't used `OFF_HEAP` mode yet) and add the unified resource release mechanism and `OFF_HEAP` mode support in another pr. This may help to make the current pr merge faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r797366940
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -66,20 +57,18 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
}
ByteBuffer buffer;
ByteBufferOutputWriter outputWriter;
- if (memoryMode == MemoryMode.OFF_HEAP) {
- outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
- } else {
- outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
- }
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
int length;
for (int i = 0; i < total; i++) {
length = lengthsVector.getInt(rowId + i);
- try {
- buffer = in.slice(length);
- } catch (EOFException e) {
- throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ if (length > 0) {
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
}
- outputWriter.write(c, rowId + i, buffer, length);
currentRow++;
Review comment:
should we change line72 to `currentRow += currentRow + total` and move it out of the looping?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -66,20 +57,18 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
}
ByteBuffer buffer;
ByteBufferOutputWriter outputWriter;
- if (memoryMode == MemoryMode.OFF_HEAP) {
- outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
- } else {
- outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
- }
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
int length;
for (int i = 0; i < total; i++) {
length = lengthsVector.getInt(rowId + i);
- try {
- buffer = in.slice(length);
- } catch (EOFException e) {
- throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ if (length > 0) {
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
}
- outputWriter.write(c, rowId + i, buffer, length);
currentRow++;
Review comment:
should we change line72 to `currentRow += currentRow + total` and move it out of the loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r804331862
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 10153 10161 12 1.5 645.5 1.0X
+SQL Json 8463 8512 69 1.9 538.0 1.2X
+SQL Parquet Vectorized: DataPageV1 131 149 14 120.0 8.3 77.5X
+SQL Parquet Vectorized: DataPageV2 98 112 15 161.2 6.2 104.0X
+SQL Parquet MR: DataPageV1 1968 1968 0 8.0 125.1 5.2X
+SQL Parquet MR: DataPageV2 1735 1739 6 9.1 110.3 5.9X
+SQL ORC Vectorized 164 198 41 96.0 10.4 62.0X
+SQL ORC MR 1572 1581 12 10.0 100.0 6.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 102 107 14 154.6 6.5 1.0X
+ParquetReader Vectorized: DataPageV2 83 88 10 189.1 5.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 57 59 3 275.7 3.6 1.8X
+ParquetReader Vectorized -> Row: DataPageV2 38 40 3 416.3 2.4 2.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12263 12285 31 1.3 779.6 1.0X
+SQL Json 9495 9501 9 1.7 603.6 1.3X
+SQL Parquet Vectorized: DataPageV1 162 175 10 97.1 10.3 75.7X
+SQL Parquet Vectorized: DataPageV2 161 172 12 97.9 10.2 76.4X
+SQL Parquet MR: DataPageV1 2074 2105 44 7.6 131.9 5.9X
+SQL Parquet MR: DataPageV2 1974 1981 9 8.0 125.5 6.2X
+SQL ORC Vectorized 187 218 30 84.3 11.9 65.7X
+SQL ORC MR 1529 1553 34 10.3 97.2 8.0X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 205 214 12 76.8 13.0 1.0X
+ParquetReader Vectorized: DataPageV2 204 211 10 77.2 13.0 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 122 132 24 128.7 7.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 122 126 6 128.4 7.8 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12946 12955 12 1.2 823.1 1.0X
+SQL Json 9871 9876 8 1.6 627.6 1.3X
+SQL Parquet Vectorized: DataPageV1 157 200 34 100.0 10.0 82.3X
+SQL Parquet Vectorized: DataPageV2 229 242 14 68.8 14.5 56.7X
+SQL Parquet MR: DataPageV1 2388 2389 2 6.6 151.8 5.4X
+SQL Parquet MR: DataPageV2 2080 2087 10 7.6 132.2 6.2X
+SQL ORC Vectorized 240 285 23 65.6 15.2 54.0X
+SQL ORC MR 1699 1732 46 9.3 108.0 7.6X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 242 245 5 65.0 15.4 1.0X
+ParquetReader Vectorized: DataPageV2 309 314 9 50.9 19.7 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 227 268 18 69.3 14.4 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 294 312 25 53.5 18.7 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14331 14347 22 1.1 911.2 1.0X
+SQL Json 10406 10434 40 1.5 661.6 1.4X
+SQL Parquet Vectorized: DataPageV1 153 196 41 102.7 9.7 93.6X
+SQL Parquet Vectorized: DataPageV2 378 415 30 41.6 24.0 37.9X
+SQL Parquet MR: DataPageV1 2439 2446 11 6.4 155.1 5.9X
+SQL Parquet MR: DataPageV2 2181 2188 10 7.2 138.7 6.6X
+SQL ORC Vectorized 320 346 25 49.2 20.3 44.8X
+SQL ORC MR 1851 1853 3 8.5 117.7 7.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 258 262 9 60.9 16.4 1.0X
+ParquetReader Vectorized: DataPageV2 481 484 3 32.7 30.6 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 250 275 26 62.9 15.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 475 502 27 33.1 30.2 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 18450 18451 2 0.9 1173.0 1.0X
+SQL Json 12553 12562 13 1.3 798.1 1.5X
+SQL Parquet Vectorized: DataPageV1 259 272 12 60.8 16.5 71.3X
+SQL Parquet Vectorized: DataPageV2 534 566 22 29.4 34.0 34.5X
+SQL Parquet MR: DataPageV1 2529 2537 11 6.2 160.8 7.3X
+SQL Parquet MR: DataPageV2 2331 2334 4 6.7 148.2 7.9X
+SQL ORC Vectorized 424 460 36 37.1 27.0 43.5X
+SQL ORC MR 2009 2023 20 7.8 127.7 9.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 322 338 21 48.9 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 674 683 12 23.3 42.9 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 352 358 9 44.7 22.4 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 628 660 22 25.0 39.9 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14806 14816 14 1.1 941.3 1.0X
+SQL Json 11968 11969 1 1.3 760.9 1.2X
+SQL Parquet Vectorized: DataPageV1 150 184 26 105.0 9.5 98.8X
+SQL Parquet Vectorized: DataPageV2 147 183 32 107.2 9.3 100.9X
+SQL Parquet MR: DataPageV1 2338 2352 19 6.7 148.7 6.3X
+SQL Parquet MR: DataPageV2 2221 2267 65 7.1 141.2 6.7X
+SQL ORC Vectorized 475 494 29 33.1 30.2 31.1X
+SQL ORC MR 1967 1978 16 8.0 125.1 7.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 236 241 8 66.7 15.0 1.0X
+ParquetReader Vectorized: DataPageV2 237 241 9 66.3 15.1 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 218 244 25 72.1 13.9 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 218 251 21 72.2 13.8 1.1X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 19335 19346 15 0.8 1229.3 1.0X
+SQL Json 16112 16121 13 1.0 1024.4 1.2X
+SQL Parquet Vectorized: DataPageV1 257 278 29 61.1 16.4 75.1X
+SQL Parquet Vectorized: DataPageV2 258 268 9 60.9 16.4 74.9X
+SQL Parquet MR: DataPageV1 2542 2557 20 6.2 161.6 7.6X
+SQL Parquet MR: DataPageV2 2416 2439 32 6.5 153.6 8.0X
+SQL ORC Vectorized 593 605 18 26.5 37.7 32.6X
+SQL ORC MR 2134 2141 11 7.4 135.7 9.1X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 322 346 24 48.8 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 326 326 1 48.3 20.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 350 359 9 44.9 22.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 348 358 10 45.2 22.1 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 13899 14000 142 0.8 1325.5 1.0X
+SQL Json 11275 11289 20 0.9 1075.3 1.2X
+SQL Parquet Vectorized: DataPageV1 2092 2107 21 5.0 199.5 6.6X
+SQL Parquet Vectorized: DataPageV2 3073 3074 2 3.4 293.0 4.5X
+SQL Parquet MR: DataPageV1 4192 4212 29 2.5 399.8 3.3X
+SQL Parquet MR: DataPageV2 4133 4194 87 2.5 394.1 3.4X
+SQL ORC Vectorized 2218 2219 1 4.7 211.5 6.3X
+SQL ORC MR 3767 3776 12 2.8 359.3 3.7X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 7367 7387 28 1.4 702.6 1.0X
+SQL Json 6817 6817 0 1.5 650.1 1.1X
+SQL Parquet Vectorized: DataPageV1 602 618 15 17.4 57.5 12.2X
+SQL Parquet Vectorized: DataPageV2 599 610 15 17.5 57.1 12.3X
+SQL Parquet MR: DataPageV1 1888 1936 68 5.6 180.0 3.9X
+SQL Parquet MR: DataPageV2 2000 2018 25 5.2 190.7 3.7X
+SQL ORC Vectorized 527 545 22 19.9 50.2 14.0X
+SQL ORC MR 1916 1927 16 5.5 182.7 3.8X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 19239 19257 25 0.8 1223.2 1.0X
+Data column - Json 12387 12393 8 1.3 787.6 1.6X
+Data column - Parquet Vectorized: DataPageV1 227 269 25 69.2 14.5 84.6X
+Data column - Parquet Vectorized: DataPageV2 612 651 28 25.7 38.9 31.4X
+Data column - Parquet MR: DataPageV1 2989 3016 39 5.3 190.0 6.4X
+Data column - Parquet MR: DataPageV2 2750 2754 5 5.7 174.8 7.0X
+Data column - ORC Vectorized 426 467 33 37.0 27.1 45.2X
+Data column - ORC MR 2513 2538 35 6.3 159.8 7.7X
+Partition column - CSV 6623 6627 5 2.4 421.1 2.9X
+Partition column - Json 10234 10235 2 1.5 650.7 1.9X
+Partition column - Parquet Vectorized: DataPageV1 56 73 19 279.8 3.6 342.2X
+Partition column - Parquet Vectorized: DataPageV2 57 72 19 278.3 3.6 340.4X
+Partition column - Parquet MR: DataPageV1 1392 1417 36 11.3 88.5 13.8X
+Partition column - Parquet MR: DataPageV2 1390 1416 37 11.3 88.4 13.8X
+Partition column - ORC Vectorized 56 89 36 283.2 3.5 346.4X
+Partition column - ORC MR 1578 1581 4 10.0 100.4 12.2X
+Both columns - CSV 19178 19181 4 0.8 1219.3 1.0X
+Both columns - Json 13104 13105 1 1.2 833.1 1.5X
+Both columns - Parquet Vectorized: DataPageV1 314 338 21 50.2 19.9 61.4X
+Both columns - Parquet Vectorized: DataPageV2 708 741 54 22.2 45.0 27.2X
+Both columns - Parquet MR: DataPageV1 3083 3105 31 5.1 196.0 6.2X
+Both columns - Parquet MR: DataPageV2 2897 2901 6 5.4 184.2 6.6X
+Both columns - ORC Vectorized 456 504 39 34.5 29.0 42.1X
+Both columns - ORC MR 2594 2597 4 6.1 164.9 7.4X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 9165 9257 130 1.1 874.1 1.0X
+SQL Json 10230 10234 7 1.0 975.6 0.9X
+SQL Parquet Vectorized: DataPageV1 1275 1315 56 8.2 121.6 7.2X
+SQL Parquet Vectorized: DataPageV2 2406 2407 0 4.4 229.5 3.8X
+SQL Parquet MR: DataPageV1 4005 4009 7 2.6 381.9 2.3X
+SQL Parquet MR: DataPageV2 4358 4366 12 2.4 415.6 2.1X
+ParquetReader Vectorized: DataPageV1 985 995 13 10.6 94.0 9.3X
+ParquetReader Vectorized: DataPageV2 2039 2061 32 5.1 194.4 4.5X
+SQL ORC Vectorized 1048 1072 34 10.0 99.9 8.7X
+SQL ORC MR 3179 3196 24 3.3 303.2 2.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 6749 6750 2 1.6 643.6 1.0X
+SQL Json 7960 7967 10 1.3 759.1 0.8X
+SQL Parquet Vectorized: DataPageV1 1078 1105 37 9.7 102.8 6.3X
+SQL Parquet Vectorized: DataPageV2 1939 1941 3 5.4 184.9 3.5X
+SQL Parquet MR: DataPageV1 3090 3099 13 3.4 294.7 2.2X
+SQL Parquet MR: DataPageV2 3274 3286 17 3.2 312.3 2.1X
+ParquetReader Vectorized: DataPageV1 1058 1067 13 9.9 100.9 6.4X
+ParquetReader Vectorized: DataPageV2 1847 1848 2 5.7 176.2 3.7X
+SQL ORC Vectorized 1307 1307 0 8.0 124.6 5.2X
+SQL ORC MR 3078 3122 62 3.4 293.6 2.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 4818 4824 8 2.2 459.5 1.0X
+SQL Json 4853 4878 35 2.2 462.8 1.0X
+SQL Parquet Vectorized: DataPageV1 255 264 6 41.1 24.3 18.9X
+SQL Parquet Vectorized: DataPageV2 711 716 4 14.7 67.8 6.8X
Review comment:
Amazing !!! This is a case worth analyzing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1085051863
Merged to master/3.3, thanks @parthchandra and all!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1061141261
@sunchao I merged your changes into the PR. Also updated the benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao closed pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao closed pull request #35262:
URL: https://github.com/apache/spark/pull/35262
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r806497889
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Got it, agree with @sunchao
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 10153 10161 12 1.5 645.5 1.0X
+SQL Json 8463 8512 69 1.9 538.0 1.2X
+SQL Parquet Vectorized: DataPageV1 131 149 14 120.0 8.3 77.5X
+SQL Parquet Vectorized: DataPageV2 98 112 15 161.2 6.2 104.0X
+SQL Parquet MR: DataPageV1 1968 1968 0 8.0 125.1 5.2X
+SQL Parquet MR: DataPageV2 1735 1739 6 9.1 110.3 5.9X
+SQL ORC Vectorized 164 198 41 96.0 10.4 62.0X
+SQL ORC MR 1572 1581 12 10.0 100.0 6.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 102 107 14 154.6 6.5 1.0X
+ParquetReader Vectorized: DataPageV2 83 88 10 189.1 5.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 57 59 3 275.7 3.6 1.8X
+ParquetReader Vectorized -> Row: DataPageV2 38 40 3 416.3 2.4 2.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12263 12285 31 1.3 779.6 1.0X
+SQL Json 9495 9501 9 1.7 603.6 1.3X
+SQL Parquet Vectorized: DataPageV1 162 175 10 97.1 10.3 75.7X
+SQL Parquet Vectorized: DataPageV2 161 172 12 97.9 10.2 76.4X
+SQL Parquet MR: DataPageV1 2074 2105 44 7.6 131.9 5.9X
+SQL Parquet MR: DataPageV2 1974 1981 9 8.0 125.5 6.2X
+SQL ORC Vectorized 187 218 30 84.3 11.9 65.7X
+SQL ORC MR 1529 1553 34 10.3 97.2 8.0X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 205 214 12 76.8 13.0 1.0X
+ParquetReader Vectorized: DataPageV2 204 211 10 77.2 13.0 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 122 132 24 128.7 7.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 122 126 6 128.4 7.8 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12946 12955 12 1.2 823.1 1.0X
+SQL Json 9871 9876 8 1.6 627.6 1.3X
+SQL Parquet Vectorized: DataPageV1 157 200 34 100.0 10.0 82.3X
+SQL Parquet Vectorized: DataPageV2 229 242 14 68.8 14.5 56.7X
+SQL Parquet MR: DataPageV1 2388 2389 2 6.6 151.8 5.4X
+SQL Parquet MR: DataPageV2 2080 2087 10 7.6 132.2 6.2X
+SQL ORC Vectorized 240 285 23 65.6 15.2 54.0X
+SQL ORC MR 1699 1732 46 9.3 108.0 7.6X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 242 245 5 65.0 15.4 1.0X
+ParquetReader Vectorized: DataPageV2 309 314 9 50.9 19.7 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 227 268 18 69.3 14.4 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 294 312 25 53.5 18.7 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14331 14347 22 1.1 911.2 1.0X
+SQL Json 10406 10434 40 1.5 661.6 1.4X
+SQL Parquet Vectorized: DataPageV1 153 196 41 102.7 9.7 93.6X
+SQL Parquet Vectorized: DataPageV2 378 415 30 41.6 24.0 37.9X
+SQL Parquet MR: DataPageV1 2439 2446 11 6.4 155.1 5.9X
+SQL Parquet MR: DataPageV2 2181 2188 10 7.2 138.7 6.6X
+SQL ORC Vectorized 320 346 25 49.2 20.3 44.8X
+SQL ORC MR 1851 1853 3 8.5 117.7 7.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 258 262 9 60.9 16.4 1.0X
+ParquetReader Vectorized: DataPageV2 481 484 3 32.7 30.6 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 250 275 26 62.9 15.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 475 502 27 33.1 30.2 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 18450 18451 2 0.9 1173.0 1.0X
+SQL Json 12553 12562 13 1.3 798.1 1.5X
+SQL Parquet Vectorized: DataPageV1 259 272 12 60.8 16.5 71.3X
+SQL Parquet Vectorized: DataPageV2 534 566 22 29.4 34.0 34.5X
+SQL Parquet MR: DataPageV1 2529 2537 11 6.2 160.8 7.3X
+SQL Parquet MR: DataPageV2 2331 2334 4 6.7 148.2 7.9X
+SQL ORC Vectorized 424 460 36 37.1 27.0 43.5X
+SQL ORC MR 2009 2023 20 7.8 127.7 9.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 322 338 21 48.9 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 674 683 12 23.3 42.9 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 352 358 9 44.7 22.4 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 628 660 22 25.0 39.9 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14806 14816 14 1.1 941.3 1.0X
+SQL Json 11968 11969 1 1.3 760.9 1.2X
+SQL Parquet Vectorized: DataPageV1 150 184 26 105.0 9.5 98.8X
+SQL Parquet Vectorized: DataPageV2 147 183 32 107.2 9.3 100.9X
+SQL Parquet MR: DataPageV1 2338 2352 19 6.7 148.7 6.3X
+SQL Parquet MR: DataPageV2 2221 2267 65 7.1 141.2 6.7X
+SQL ORC Vectorized 475 494 29 33.1 30.2 31.1X
+SQL ORC MR 1967 1978 16 8.0 125.1 7.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 236 241 8 66.7 15.0 1.0X
+ParquetReader Vectorized: DataPageV2 237 241 9 66.3 15.1 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 218 244 25 72.1 13.9 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 218 251 21 72.2 13.8 1.1X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 19335 19346 15 0.8 1229.3 1.0X
+SQL Json 16112 16121 13 1.0 1024.4 1.2X
+SQL Parquet Vectorized: DataPageV1 257 278 29 61.1 16.4 75.1X
+SQL Parquet Vectorized: DataPageV2 258 268 9 60.9 16.4 74.9X
+SQL Parquet MR: DataPageV1 2542 2557 20 6.2 161.6 7.6X
+SQL Parquet MR: DataPageV2 2416 2439 32 6.5 153.6 8.0X
+SQL ORC Vectorized 593 605 18 26.5 37.7 32.6X
+SQL ORC MR 2134 2141 11 7.4 135.7 9.1X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 322 346 24 48.8 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 326 326 1 48.3 20.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 350 359 9 44.9 22.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 348 358 10 45.2 22.1 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 13899 14000 142 0.8 1325.5 1.0X
+SQL Json 11275 11289 20 0.9 1075.3 1.2X
+SQL Parquet Vectorized: DataPageV1 2092 2107 21 5.0 199.5 6.6X
+SQL Parquet Vectorized: DataPageV2 3073 3074 2 3.4 293.0 4.5X
+SQL Parquet MR: DataPageV1 4192 4212 29 2.5 399.8 3.3X
+SQL Parquet MR: DataPageV2 4133 4194 87 2.5 394.1 3.4X
+SQL ORC Vectorized 2218 2219 1 4.7 211.5 6.3X
+SQL ORC MR 3767 3776 12 2.8 359.3 3.7X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 7367 7387 28 1.4 702.6 1.0X
+SQL Json 6817 6817 0 1.5 650.1 1.1X
+SQL Parquet Vectorized: DataPageV1 602 618 15 17.4 57.5 12.2X
+SQL Parquet Vectorized: DataPageV2 599 610 15 17.5 57.1 12.3X
+SQL Parquet MR: DataPageV1 1888 1936 68 5.6 180.0 3.9X
+SQL Parquet MR: DataPageV2 2000 2018 25 5.2 190.7 3.7X
+SQL ORC Vectorized 527 545 22 19.9 50.2 14.0X
+SQL ORC MR 1916 1927 16 5.5 182.7 3.8X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 19239 19257 25 0.8 1223.2 1.0X
+Data column - Json 12387 12393 8 1.3 787.6 1.6X
+Data column - Parquet Vectorized: DataPageV1 227 269 25 69.2 14.5 84.6X
+Data column - Parquet Vectorized: DataPageV2 612 651 28 25.7 38.9 31.4X
+Data column - Parquet MR: DataPageV1 2989 3016 39 5.3 190.0 6.4X
+Data column - Parquet MR: DataPageV2 2750 2754 5 5.7 174.8 7.0X
+Data column - ORC Vectorized 426 467 33 37.0 27.1 45.2X
+Data column - ORC MR 2513 2538 35 6.3 159.8 7.7X
+Partition column - CSV 6623 6627 5 2.4 421.1 2.9X
+Partition column - Json 10234 10235 2 1.5 650.7 1.9X
+Partition column - Parquet Vectorized: DataPageV1 56 73 19 279.8 3.6 342.2X
+Partition column - Parquet Vectorized: DataPageV2 57 72 19 278.3 3.6 340.4X
+Partition column - Parquet MR: DataPageV1 1392 1417 36 11.3 88.5 13.8X
+Partition column - Parquet MR: DataPageV2 1390 1416 37 11.3 88.4 13.8X
+Partition column - ORC Vectorized 56 89 36 283.2 3.5 346.4X
+Partition column - ORC MR 1578 1581 4 10.0 100.4 12.2X
+Both columns - CSV 19178 19181 4 0.8 1219.3 1.0X
+Both columns - Json 13104 13105 1 1.2 833.1 1.5X
+Both columns - Parquet Vectorized: DataPageV1 314 338 21 50.2 19.9 61.4X
+Both columns - Parquet Vectorized: DataPageV2 708 741 54 22.2 45.0 27.2X
+Both columns - Parquet MR: DataPageV1 3083 3105 31 5.1 196.0 6.2X
+Both columns - Parquet MR: DataPageV2 2897 2901 6 5.4 184.2 6.6X
+Both columns - ORC Vectorized 456 504 39 34.5 29.0 42.1X
+Both columns - ORC MR 2594 2597 4 6.1 164.9 7.4X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 9165 9257 130 1.1 874.1 1.0X
+SQL Json 10230 10234 7 1.0 975.6 0.9X
+SQL Parquet Vectorized: DataPageV1 1275 1315 56 8.2 121.6 7.2X
+SQL Parquet Vectorized: DataPageV2 2406 2407 0 4.4 229.5 3.8X
+SQL Parquet MR: DataPageV1 4005 4009 7 2.6 381.9 2.3X
+SQL Parquet MR: DataPageV2 4358 4366 12 2.4 415.6 2.1X
+ParquetReader Vectorized: DataPageV1 985 995 13 10.6 94.0 9.3X
+ParquetReader Vectorized: DataPageV2 2039 2061 32 5.1 194.4 4.5X
+SQL ORC Vectorized 1048 1072 34 10.0 99.9 8.7X
+SQL ORC MR 3179 3196 24 3.3 303.2 2.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 6749 6750 2 1.6 643.6 1.0X
+SQL Json 7960 7967 10 1.3 759.1 0.8X
+SQL Parquet Vectorized: DataPageV1 1078 1105 37 9.7 102.8 6.3X
+SQL Parquet Vectorized: DataPageV2 1939 1941 3 5.4 184.9 3.5X
+SQL Parquet MR: DataPageV1 3090 3099 13 3.4 294.7 2.2X
+SQL Parquet MR: DataPageV2 3274 3286 17 3.2 312.3 2.1X
+ParquetReader Vectorized: DataPageV1 1058 1067 13 9.9 100.9 6.4X
+ParquetReader Vectorized: DataPageV2 1847 1848 2 5.7 176.2 3.7X
+SQL ORC Vectorized 1307 1307 0 8.0 124.6 5.2X
+SQL ORC MR 3078 3122 62 3.4 293.6 2.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 4818 4824 8 2.2 459.5 1.0X
+SQL Json 4853 4878 35 2.2 462.8 1.0X
+SQL Parquet Vectorized: DataPageV1 255 264 6 41.1 24.3 18.9X
+SQL Parquet Vectorized: DataPageV2 711 716 4 14.7 67.8 6.8X
Review comment:
After [406d176](https://github.com/apache/spark/pull/35262/commits/406d1768019eed1516925847a59bd2ad5fc7883b), should we update bench result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1041069155
> Updated the benchmarks.
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r805072581
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
☝️ @LuciferYang , @sunchao, should I remove this or not?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
Review comment:
Removed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
Review comment:
Also, I'm running the benchmarks again. Will update the results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r797365496
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -66,20 +57,18 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
}
ByteBuffer buffer;
ByteBufferOutputWriter outputWriter;
- if (memoryMode == MemoryMode.OFF_HEAP) {
- outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
- } else {
- outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
- }
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
Review comment:
can merge lines 59 and 60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1029295384
Thank you for your explanation @parthchandra ,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r825195773
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,50 +16,126 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader;
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private ByteBuffer previous;
+ private int currentRow = 0;
+
+ // temporary variable used by getBinary
Review comment:
nit: `getBinary` -> `readBinary`. Also can we add some comments on what `tempBinaryValVector` is for?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -443,6 +444,8 @@ public UTF8String getUTF8String(int rowId) {
}
}
+ public abstract ByteBuffer getBytesUnsafe(int rowId, int count);
Review comment:
nit: maybe add a few comments here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789351038
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -86,4 +87,26 @@ void readLongsWithRebase(
void write(WritableColumnVector outputColumnVector, int rowId, long val);
}
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
+ void write(WritableColumnVector c, int rowId, ByteBuffer val, int length);
+
+ static void writeArrayByteBuffer(WritableColumnVector c, int rowId, ByteBuffer val,
Review comment:
Is it a good practice to add `static` methods to interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r790517761
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -147,12 +147,19 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
withTempPath { dir =>
val path = s"${dir.getCanonicalPath}/test.parquet"
- val data = (1 to 3).map { i =>
- ( i, i.toLong, i.toShort, Array[Byte](i.toByte), s"test_${i}",
- DateTimeUtils.fromJavaDate(Date.valueOf(s"2021-11-0" + i)),
- DateTimeUtils.fromJavaTimestamp(Timestamp.valueOf(s"2020-11-01 12:00:0" + i)),
- Period.of(1, i, 0), Duration.ofMillis(i * 100),
- new BigDecimal(java.lang.Long.toUnsignedString(i*100000))
+ // Have more than 2 * 4096 records (so we have multiple tasks and each task
+ // reads at least twice from the reader). This will catch any issues with state
+ // maintained by the reader(s)
+ // Add at least one string with a null
+ val data = (1 to 8197).map { i =>
+ ( i,
+ i.toLong, i.toShort, Array[Byte](i.toByte),
+ if (i % 2 == 1) s"test_${i}" else null,
Review comment:
test_$i
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r791231334
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -147,12 +147,19 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
withTempPath { dir =>
val path = s"${dir.getCanonicalPath}/test.parquet"
- val data = (1 to 3).map { i =>
- ( i, i.toLong, i.toShort, Array[Byte](i.toByte), s"test_${i}",
- DateTimeUtils.fromJavaDate(Date.valueOf(s"2021-11-0" + i)),
- DateTimeUtils.fromJavaTimestamp(Timestamp.valueOf(s"2020-11-01 12:00:0" + i)),
- Period.of(1, i, 0), Duration.ofMillis(i * 100),
- new BigDecimal(java.lang.Long.toUnsignedString(i*100000))
+ // Have more than 2 * 4096 records (so we have multiple tasks and each task
+ // reads at least twice from the reader). This will catch any issues with state
+ // maintained by the reader(s)
+ // Add at least one string with a null
+ val data = (1 to 8197).map { i =>
+ ( i,
+ i.toLong, i.toShort, Array[Byte](i.toByte),
+ if (i % 2 == 1) s"test_${i}" else null,
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ prefixLengthVector = new OffHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OffHeapColumnVector(valueCount, BinaryType);
+ } else {
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ }
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, null, 0,
+ (w, r, v, l) ->
Review comment:
What you say makes sense: that the reference to the external variable will cause multiple object instantiation. Thank you for doing this reasearch!
I tried something similar but with the unit test in ParquetEncodingSuite and see only a single instance of the lambda created (not sure why).
I've changed the code to use a WritableVector of size 1 which eliminates the need to access the variable directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r791326157
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
Review comment:
this `valueCount` can also be removed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,6 +96,12 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
HadoopInputFile.fromPath(file, configuration), options);
this.reader = new ParquetRowGroupReaderImpl(fileReader);
this.fileSchema = fileReader.getFileMetaData().getSchema();
+ try {
+ this.writerVersion = VersionParser.parse(fileReader.getFileMetaData().getCreatedBy());
+ } catch (Exception e) {
Review comment:
Will other types of exceptions be thrown here, except `VersionParseException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r793451912
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
Review comment:
Maybe we should call `lengthsVector.putInts(0, valueCount, 0);` to ensure this init value of OffHeapColumnVector, or use other means to avoid reading unexpected values in `line 75` when `memoryMode is MemoryMode.OFF_HEAP`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794201795
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
null values will not be handled here since they are represented using definition levels in Parquet. I think only empty strings matter here but it's likely not so common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794168786
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
so if `length` is 0, can we simplify the operation of `line77 ~ line82`, I'm not sure if this will optimize the scene with a large number of null values, such as `String with Nulls Scan`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794236177
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
@parthchandra if `This method gets called only for non-null values.`, then there should be no need to add `lengthsVector.putInts(0, valueCount, 0)` because the place where the exception was thrown was originally a null value when running `String with Nulls Scan`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789323003
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
Review comment:
should private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789342727
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ prefixLengthVector = new OffHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OffHeapColumnVector(valueCount, BinaryType);
+ } else {
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ }
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, null, 0,
+ (w, r, v, l) ->
Review comment:
Does this lambda will cause frequent object creation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789342727
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ prefixLengthVector = new OffHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OffHeapColumnVector(valueCount, BinaryType);
+ } else {
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ }
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, null, 0,
+ (w, r, v, l) ->
Review comment:
Will this lambda create a new object every time when `readBinary` is called?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r795333708
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
So I think it may be valuable to avoid some unnecessary operations when length is 0. At the same time, this may be related to the performance degradation in `String with Nulls Scan(95%)` scenario, because 95% of the values in this scenario are values with length of 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1026231119
It is generally a good idea to use off-heap memory if the writable vector is off-heap as copying data between off-heap and on-heap memory is somewhat more expensive. But since it is leading to the memory leak, it is a good idea to remove the off heap usage. I didn't see a difference in performance, so perhaps the effect in this case is not that great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sadikovi commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r827612677
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -283,6 +290,11 @@ private void initDataReader(
} catch (IOException e) {
throw new IOException("could not read page in col " + descriptor, e);
}
+ if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding) &&
+ previousReader instanceof RequiresPreviousReader) {
+ // previous reader can only be set if reading sequentially
Review comment:
nit: [P]revious.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
##########
@@ -90,13 +90,18 @@ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOExce
Preconditions.checkArgument(miniSize % 8 == 0,
"miniBlockSize must be multiple of 8, but it's " + miniSize);
this.miniBlockSizeInValues = (int) miniSize;
+ // True value count. May be less than valueCount because of nulls
Review comment:
I think it would be more useful to annotate the method `getTotalValueCount` instead of here.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -283,6 +290,11 @@ private void initDataReader(
} catch (IOException e) {
throw new IOException("could not read page in col " + descriptor, e);
}
+ if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding) &&
Review comment:
When does this happen? Can you add a comment on why we need this?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,50 +16,127 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader;
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private ByteBuffer previous;
+ private int currentRow = 0;
+
+ // temporary variable used by readBinary
+ private final WritableColumnVector binaryValVector;
+ // temporary variable used by skipBinary
Review comment:
nit: Upper case.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -17,6 +17,7 @@
package org.apache.spark.sql.execution.datasources.parquet;
+import java.nio.ByteBuffer;
Review comment:
nit: new line after the import.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaByteArrayEncodingSuite.scala
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.parquet.bytes.DirectByteBufferAllocator
+import org.apache.parquet.column.values.Utils
+import org.apache.parquet.column.values.deltastrings.DeltaByteArrayWriter
+
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{IntegerType, StringType}
+
+/**
+ * Read tests for vectorized Delta byte array reader.
+ * Translated from * org.apache.parquet.column.values.delta.TestDeltaByteArray
+ */
+class ParquetDeltaByteArrayEncodingSuite extends ParquetCompatibilityTest with SharedSparkSession {
+ val values: Array[String] = Array("parquet-mr", "parquet", "parquet-format");
+ val randvalues: Array[String] = Utils.getRandomStringSamples(10000, 32)
+
+ var writer: DeltaByteArrayWriter = _
+ var reader: VectorizedDeltaByteArrayReader = _
+ private var writableColumnVector: WritableColumnVector = _
+
+ protected override def beforeEach(): Unit = {
+ writer = new DeltaByteArrayWriter(64 * 1024, 64 * 1024, new DirectByteBufferAllocator)
+ reader = new VectorizedDeltaByteArrayReader()
+ super.beforeAll()
+ }
+
+ test("test Serialization") {
+ assertReadWrite(writer, reader, values)
+ }
+
+ test("random strings") {
+ assertReadWrite(writer, reader, randvalues)
+ }
+
+ test("random strings with skip") {
+ assertReadWriteWithSkip(writer, reader, randvalues)
+ }
+
+ test("random strings with skipN") {
+ assertReadWriteWithSkipN(writer, reader, randvalues)
+ }
+
+ test("test lengths") {
+ var reader = new VectorizedDeltaBinaryPackedReader
+ Utils.writeData(writer, values)
+ val data = writer.getBytes.toInputStream
+ val length = values.length
+ writableColumnVector = new OnHeapColumnVector(length, IntegerType)
+ reader.initFromPage(length, data)
+ reader.readIntegers(length, writableColumnVector, 0)
+ // test prefix lengths
+ assert(0 == writableColumnVector.getInt(0))
+ assert(7 == writableColumnVector.getInt(1))
+ assert(7 == writableColumnVector.getInt(2))
+
+ reader = new VectorizedDeltaBinaryPackedReader
+ writableColumnVector = new OnHeapColumnVector(length, IntegerType)
+ reader.initFromPage(length, data)
+ reader.readIntegers(length, writableColumnVector, 0)
+ // test suffix lengths
+ assert(10 == writableColumnVector.getInt(0))
+ assert(0 == writableColumnVector.getInt(1))
+ assert(7 == writableColumnVector.getInt(2))
+ }
+
+ private def assertReadWrite(
+ writer: DeltaByteArrayWriter,
+ reader: VectorizedDeltaByteArrayReader,
+ vals: Array[String]): Unit = {
+ Utils.writeData(writer, vals)
+ val length = vals.length
+ val is = writer.getBytes.toInputStream
+
+ writableColumnVector = new OnHeapColumnVector(length, StringType)
+
+ reader.initFromPage(length, is)
+ reader.readBinary(length, writableColumnVector, 0)
+
+ for (i <- 0 until length) {
+ assert(vals(i).getBytes() sameElements writableColumnVector.getBinary(i))
+ }
+ }
+
+ private def assertReadWriteWithSkip(
+ writer: DeltaByteArrayWriter,
+ reader: VectorizedDeltaByteArrayReader,
+ vals: Array[String]): Unit = {
+ Utils.writeData(writer, vals)
+ val length = vals.length
+ val is = writer.getBytes.toInputStream
+ writableColumnVector = new OnHeapColumnVector(length, StringType)
+ reader.initFromPage(length, is)
+ var i = 0
+ while ( {
+ i < vals.length
+ }) {
+ reader.readBinary(1, writableColumnVector, i)
+ assert(vals(i).getBytes() sameElements writableColumnVector.getBinary(i))
+ reader.skipBinary(1)
+ i += 2
+ }
+ }
+
+ private def assertReadWriteWithSkipN(
+ writer: DeltaByteArrayWriter,
+ reader: VectorizedDeltaByteArrayReader,
+ vals: Array[String]): Unit = {
+ Utils.writeData(writer, vals)
+ val length = vals.length
+ val is = writer.getBytes.toInputStream
+ writableColumnVector = new OnHeapColumnVector(length, StringType)
+ reader.initFromPage(length, is)
+ var skipCount = 0
+ var i = 0
+ while ( {
+ i < vals.length
+ }) {
+ skipCount = (vals.length - i) / 2
+ reader.readBinary(1, writableColumnVector, i)
+ assert(vals(i).getBytes() sameElements writableColumnVector.getBinary(i))
+ reader.skipBinary(skipCount)
+ i += skipCount + 1
+ }
+ }
+
Review comment:
ditto.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -443,6 +444,8 @@ public UTF8String getUTF8String(int rowId) {
}
}
+ public abstract ByteBuffer getBytesUnsafe(int rowId, int count);
Review comment:
I agree, the method is misleading since there is a memory copy involved, it is just does not call System.arraycopy in OnHeapColumnVector.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,50 +16,127 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader;
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private ByteBuffer previous;
+ private int currentRow = 0;
+
+ // temporary variable used by readBinary
Review comment:
nit: Upper case.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -141,45 +149,54 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
)
val hadoopConf = spark.sessionState.newHadoopConfWithOptions(extraOptions)
- withSQLConf(
- SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",
- ParquetOutputFormat.JOB_SUMMARY_LEVEL -> "ALL") {
- withTempPath { dir =>
- val path = s"${dir.getCanonicalPath}/test.parquet"
-
- val data = (1 to 3).map { i =>
- ( i, i.toLong, i.toShort, Array[Byte](i.toByte), s"test_${i}",
- DateTimeUtils.fromJavaDate(Date.valueOf(s"2021-11-0" + i)),
- DateTimeUtils.fromJavaTimestamp(Timestamp.valueOf(s"2020-11-01 12:00:0" + i)),
- Period.of(1, i, 0), Duration.ofMillis(i * 100),
- new BigDecimal(java.lang.Long.toUnsignedString(i*100000))
- )
+ withMemoryModes { offHeapMode =>
+ withSQLConf(
+ SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> offHeapMode,
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",
+ ParquetOutputFormat.JOB_SUMMARY_LEVEL -> "ALL") {
+ withTempPath { dir =>
+ val path = s"${dir.getCanonicalPath}/test.parquet"
+ // Have more than 2 * 4096 records (so we have multiple tasks and each task
+ // reads at least twice from the reader). This will catch any issues with state
+ // maintained by the reader(s)
+ // Add at least one string with a null
+ val data = (1 to 81971).map { i =>
Review comment:
I don't quite understand how this number was chosen. Can you elaborate? Can we make it `2 * 4096 + 1` - would it work as well?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -86,4 +87,20 @@ void readLongsWithRebase(
void write(WritableColumnVector outputColumnVector, int rowId, long val);
}
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
+ void write(WritableColumnVector c, int rowId, ByteBuffer val, int length);
+
+ static void writeArrayByteBuffer(WritableColumnVector c, int rowId, ByteBuffer val,
+ int length) {
+ c.putByteArray(rowId,
+ val.array(),
+ val.arrayOffset() + val.position(),
+ length);
+ }
+
+ static void skipWrite(WritableColumnVector c, int rowId, ByteBuffer val, int length) { }
+
+ }
+
Review comment:
nit: new line.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
##########
@@ -367,7 +367,9 @@ private void checkEndOfRowGroup() throws IOException {
datetimeRebaseMode,
datetimeRebaseTz,
int96RebaseMode,
- int96RebaseTz);
+ int96RebaseTz,
+ writerVersion
+ );
Review comment:
nit: should be on the previous line.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java
##########
@@ -221,6 +221,13 @@ protected UTF8String getBytesAsUTF8String(int rowId, int count) {
return UTF8String.fromAddress(null, data + rowId, count);
}
+ @Override
+ public ByteBuffer getBytesUnsafe(int rowId, int count) {
Review comment:
Can we replace it with:
```java
return ByteBuffer.wrap(getBytes(rowId, count));
```
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader;
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader() {
+ lengthReader = new VectorizedDeltaBinaryPackedReader();
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
+ }
+ currentRow += total;
+ }
+
+ public ByteBuffer getBytes(int rowId) {
+ int length = lengthsVector.getInt(rowId);
+ try {
+ return in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ for (int i = 0; i < total; i++) {
+ int remaining = lengthsVector.getInt(currentRow + i);
+ while (remaining > 0) {
+ remaining -= in.skip(remaining);
+ }
+ }
+ currentRow += total;
+ }
+
Review comment:
nit: new line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1068453537
Addressed the latest few comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1085048833
Sorry for the delay. Going to merge this now since the PR is included in the the allowed list of Spark 3.3. The linter issue looks unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1085051863
Merged, thanks @parthchandra and all!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r804255899
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
+ int length = prefixLength + suffix.length;
+
+ // NOTE: due to PARQUET-246, it is important that we
+ // respect prefixLength which was read from prefixLengthReader,
+ // even for the *first* value of a page. Even though the first
+ // value of the page should have an empty prefix, it may not
+ // because of PARQUET-246.
+
+ // We have to do this to materialize the output
+ if (prefixLength != 0) {
+ // We could do
+ // c.putByteArray(rowId + i, previous, 0, prefixLength);
+ // c.putByteArray(rowId+i, suffix, prefixLength, suffix.length);
+ // previous = c.getBinary(rowId+1);
+ // but it incurs the same cost of copying the values twice _and_ c.getBinary
+ // is a _slow_ byte by byte copy
+ // The following always uses the faster system arraycopy method
+ byte[] out = new byte[length];
+ System.arraycopy(previous, 0, out, 0, prefixLength);
+ System.arraycopy(suffix, 0, out, prefixLength, suffix.length);
+ previous = out;
} else {
- byte[] bytes = new byte[binary.length()];
- buffer.get(bytes);
- c.putByteArray(rowId + i, bytes);
+ previous = suffix;
}
+ outputWriter.write(c, rowId + i, ByteBuffer.wrap(previous), previous.length);
+ currentRow++;
}
}
@Override
- public void skipBinary(int total) {
- for (int i = 0; i < total; i++) {
- deltaByteArrayReader.skip();
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, ByteBufferOutputWriter::writeArrayByteBuffer);
+ }
+
+ /**
+ * There was a bug (PARQUET-246) in which DeltaByteArrayWriter's reset() method did not clear the
+ * previous value state that it tracks internally. This resulted in the first value of all pages
Review comment:
Oh cool, it's good then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1041009200
Updated the benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r805092339
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
I think we should remove it since empty string should also have offset and length in the `WritableColumnVector`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r805092409
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
We should probably have a test case BTW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r802222581
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
Review comment:
https://github.com/apache/spark/pull/35262#issuecomment-1025369374
As discussed earlier, if we use `OffHeapColumnVector `, we need to consider the memory leak issue because there is no path to call `OffHeapColumnVector.close()` method currently, this will make the current pr more complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r800929305
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
>
> Also cc @dongjoon-hyun @viirya @sunchao for further review
@dongjoon-hyun, @viirya, @sunchao any further comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1023832057
@LuciferYang you were right, the vector needed initialization.
Also added a memory mode parameter for the unit test. I increased the size of the data generated to reproduce the issue and then applied to fix. Also ran the benchmark with the off heap enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r795289182
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
If the column encoding is `DELTA_BYTE_ARRAY` the data is in encoded as a vector of prefix lengths and a vector of suffixes. The vector of suffixes is a `DELTA_LENGTH_BYTE_ARRAY` encoding. So we may have a `DELTA_LENGTH_BYTE_ARRAY` where we have empty strings even if the value is not null. (AFAIK this is the only use of `DELTA_LENGTH_BYTE_ARRAY`).
Also, `lengthsVector` as you noted, cannot distinguish between empty string and null bin `readBinary` scope because it doesn't have the definition level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794197228
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
I'm not entirely sure. Does the WritableColumnVector allow skipping values? I assume so, but I'm not sure. Do you know?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1039934162
Can you re-trigger GA? @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r806510255
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 10153 10161 12 1.5 645.5 1.0X
+SQL Json 8463 8512 69 1.9 538.0 1.2X
+SQL Parquet Vectorized: DataPageV1 131 149 14 120.0 8.3 77.5X
+SQL Parquet Vectorized: DataPageV2 98 112 15 161.2 6.2 104.0X
+SQL Parquet MR: DataPageV1 1968 1968 0 8.0 125.1 5.2X
+SQL Parquet MR: DataPageV2 1735 1739 6 9.1 110.3 5.9X
+SQL ORC Vectorized 164 198 41 96.0 10.4 62.0X
+SQL ORC MR 1572 1581 12 10.0 100.0 6.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 102 107 14 154.6 6.5 1.0X
+ParquetReader Vectorized: DataPageV2 83 88 10 189.1 5.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 57 59 3 275.7 3.6 1.8X
+ParquetReader Vectorized -> Row: DataPageV2 38 40 3 416.3 2.4 2.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12263 12285 31 1.3 779.6 1.0X
+SQL Json 9495 9501 9 1.7 603.6 1.3X
+SQL Parquet Vectorized: DataPageV1 162 175 10 97.1 10.3 75.7X
+SQL Parquet Vectorized: DataPageV2 161 172 12 97.9 10.2 76.4X
+SQL Parquet MR: DataPageV1 2074 2105 44 7.6 131.9 5.9X
+SQL Parquet MR: DataPageV2 1974 1981 9 8.0 125.5 6.2X
+SQL ORC Vectorized 187 218 30 84.3 11.9 65.7X
+SQL ORC MR 1529 1553 34 10.3 97.2 8.0X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 205 214 12 76.8 13.0 1.0X
+ParquetReader Vectorized: DataPageV2 204 211 10 77.2 13.0 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 122 132 24 128.7 7.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 122 126 6 128.4 7.8 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12946 12955 12 1.2 823.1 1.0X
+SQL Json 9871 9876 8 1.6 627.6 1.3X
+SQL Parquet Vectorized: DataPageV1 157 200 34 100.0 10.0 82.3X
+SQL Parquet Vectorized: DataPageV2 229 242 14 68.8 14.5 56.7X
+SQL Parquet MR: DataPageV1 2388 2389 2 6.6 151.8 5.4X
+SQL Parquet MR: DataPageV2 2080 2087 10 7.6 132.2 6.2X
+SQL ORC Vectorized 240 285 23 65.6 15.2 54.0X
+SQL ORC MR 1699 1732 46 9.3 108.0 7.6X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 242 245 5 65.0 15.4 1.0X
+ParquetReader Vectorized: DataPageV2 309 314 9 50.9 19.7 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 227 268 18 69.3 14.4 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 294 312 25 53.5 18.7 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14331 14347 22 1.1 911.2 1.0X
+SQL Json 10406 10434 40 1.5 661.6 1.4X
+SQL Parquet Vectorized: DataPageV1 153 196 41 102.7 9.7 93.6X
+SQL Parquet Vectorized: DataPageV2 378 415 30 41.6 24.0 37.9X
+SQL Parquet MR: DataPageV1 2439 2446 11 6.4 155.1 5.9X
+SQL Parquet MR: DataPageV2 2181 2188 10 7.2 138.7 6.6X
+SQL ORC Vectorized 320 346 25 49.2 20.3 44.8X
+SQL ORC MR 1851 1853 3 8.5 117.7 7.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 258 262 9 60.9 16.4 1.0X
+ParquetReader Vectorized: DataPageV2 481 484 3 32.7 30.6 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 250 275 26 62.9 15.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 475 502 27 33.1 30.2 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 18450 18451 2 0.9 1173.0 1.0X
+SQL Json 12553 12562 13 1.3 798.1 1.5X
+SQL Parquet Vectorized: DataPageV1 259 272 12 60.8 16.5 71.3X
+SQL Parquet Vectorized: DataPageV2 534 566 22 29.4 34.0 34.5X
+SQL Parquet MR: DataPageV1 2529 2537 11 6.2 160.8 7.3X
+SQL Parquet MR: DataPageV2 2331 2334 4 6.7 148.2 7.9X
+SQL ORC Vectorized 424 460 36 37.1 27.0 43.5X
+SQL ORC MR 2009 2023 20 7.8 127.7 9.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 322 338 21 48.9 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 674 683 12 23.3 42.9 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 352 358 9 44.7 22.4 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 628 660 22 25.0 39.9 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14806 14816 14 1.1 941.3 1.0X
+SQL Json 11968 11969 1 1.3 760.9 1.2X
+SQL Parquet Vectorized: DataPageV1 150 184 26 105.0 9.5 98.8X
+SQL Parquet Vectorized: DataPageV2 147 183 32 107.2 9.3 100.9X
+SQL Parquet MR: DataPageV1 2338 2352 19 6.7 148.7 6.3X
+SQL Parquet MR: DataPageV2 2221 2267 65 7.1 141.2 6.7X
+SQL ORC Vectorized 475 494 29 33.1 30.2 31.1X
+SQL ORC MR 1967 1978 16 8.0 125.1 7.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 236 241 8 66.7 15.0 1.0X
+ParquetReader Vectorized: DataPageV2 237 241 9 66.3 15.1 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 218 244 25 72.1 13.9 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 218 251 21 72.2 13.8 1.1X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 19335 19346 15 0.8 1229.3 1.0X
+SQL Json 16112 16121 13 1.0 1024.4 1.2X
+SQL Parquet Vectorized: DataPageV1 257 278 29 61.1 16.4 75.1X
+SQL Parquet Vectorized: DataPageV2 258 268 9 60.9 16.4 74.9X
+SQL Parquet MR: DataPageV1 2542 2557 20 6.2 161.6 7.6X
+SQL Parquet MR: DataPageV2 2416 2439 32 6.5 153.6 8.0X
+SQL ORC Vectorized 593 605 18 26.5 37.7 32.6X
+SQL ORC MR 2134 2141 11 7.4 135.7 9.1X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 322 346 24 48.8 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 326 326 1 48.3 20.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 350 359 9 44.9 22.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 348 358 10 45.2 22.1 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 13899 14000 142 0.8 1325.5 1.0X
+SQL Json 11275 11289 20 0.9 1075.3 1.2X
+SQL Parquet Vectorized: DataPageV1 2092 2107 21 5.0 199.5 6.6X
+SQL Parquet Vectorized: DataPageV2 3073 3074 2 3.4 293.0 4.5X
+SQL Parquet MR: DataPageV1 4192 4212 29 2.5 399.8 3.3X
+SQL Parquet MR: DataPageV2 4133 4194 87 2.5 394.1 3.4X
+SQL ORC Vectorized 2218 2219 1 4.7 211.5 6.3X
+SQL ORC MR 3767 3776 12 2.8 359.3 3.7X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 7367 7387 28 1.4 702.6 1.0X
+SQL Json 6817 6817 0 1.5 650.1 1.1X
+SQL Parquet Vectorized: DataPageV1 602 618 15 17.4 57.5 12.2X
+SQL Parquet Vectorized: DataPageV2 599 610 15 17.5 57.1 12.3X
+SQL Parquet MR: DataPageV1 1888 1936 68 5.6 180.0 3.9X
+SQL Parquet MR: DataPageV2 2000 2018 25 5.2 190.7 3.7X
+SQL ORC Vectorized 527 545 22 19.9 50.2 14.0X
+SQL ORC MR 1916 1927 16 5.5 182.7 3.8X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 19239 19257 25 0.8 1223.2 1.0X
+Data column - Json 12387 12393 8 1.3 787.6 1.6X
+Data column - Parquet Vectorized: DataPageV1 227 269 25 69.2 14.5 84.6X
+Data column - Parquet Vectorized: DataPageV2 612 651 28 25.7 38.9 31.4X
+Data column - Parquet MR: DataPageV1 2989 3016 39 5.3 190.0 6.4X
+Data column - Parquet MR: DataPageV2 2750 2754 5 5.7 174.8 7.0X
+Data column - ORC Vectorized 426 467 33 37.0 27.1 45.2X
+Data column - ORC MR 2513 2538 35 6.3 159.8 7.7X
+Partition column - CSV 6623 6627 5 2.4 421.1 2.9X
+Partition column - Json 10234 10235 2 1.5 650.7 1.9X
+Partition column - Parquet Vectorized: DataPageV1 56 73 19 279.8 3.6 342.2X
+Partition column - Parquet Vectorized: DataPageV2 57 72 19 278.3 3.6 340.4X
+Partition column - Parquet MR: DataPageV1 1392 1417 36 11.3 88.5 13.8X
+Partition column - Parquet MR: DataPageV2 1390 1416 37 11.3 88.4 13.8X
+Partition column - ORC Vectorized 56 89 36 283.2 3.5 346.4X
+Partition column - ORC MR 1578 1581 4 10.0 100.4 12.2X
+Both columns - CSV 19178 19181 4 0.8 1219.3 1.0X
+Both columns - Json 13104 13105 1 1.2 833.1 1.5X
+Both columns - Parquet Vectorized: DataPageV1 314 338 21 50.2 19.9 61.4X
+Both columns - Parquet Vectorized: DataPageV2 708 741 54 22.2 45.0 27.2X
+Both columns - Parquet MR: DataPageV1 3083 3105 31 5.1 196.0 6.2X
+Both columns - Parquet MR: DataPageV2 2897 2901 6 5.4 184.2 6.6X
+Both columns - ORC Vectorized 456 504 39 34.5 29.0 42.1X
+Both columns - ORC MR 2594 2597 4 6.1 164.9 7.4X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 9165 9257 130 1.1 874.1 1.0X
+SQL Json 10230 10234 7 1.0 975.6 0.9X
+SQL Parquet Vectorized: DataPageV1 1275 1315 56 8.2 121.6 7.2X
+SQL Parquet Vectorized: DataPageV2 2406 2407 0 4.4 229.5 3.8X
+SQL Parquet MR: DataPageV1 4005 4009 7 2.6 381.9 2.3X
+SQL Parquet MR: DataPageV2 4358 4366 12 2.4 415.6 2.1X
+ParquetReader Vectorized: DataPageV1 985 995 13 10.6 94.0 9.3X
+ParquetReader Vectorized: DataPageV2 2039 2061 32 5.1 194.4 4.5X
+SQL ORC Vectorized 1048 1072 34 10.0 99.9 8.7X
+SQL ORC MR 3179 3196 24 3.3 303.2 2.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 6749 6750 2 1.6 643.6 1.0X
+SQL Json 7960 7967 10 1.3 759.1 0.8X
+SQL Parquet Vectorized: DataPageV1 1078 1105 37 9.7 102.8 6.3X
+SQL Parquet Vectorized: DataPageV2 1939 1941 3 5.4 184.9 3.5X
+SQL Parquet MR: DataPageV1 3090 3099 13 3.4 294.7 2.2X
+SQL Parquet MR: DataPageV2 3274 3286 17 3.2 312.3 2.1X
+ParquetReader Vectorized: DataPageV1 1058 1067 13 9.9 100.9 6.4X
+ParquetReader Vectorized: DataPageV2 1847 1848 2 5.7 176.2 3.7X
+SQL ORC Vectorized 1307 1307 0 8.0 124.6 5.2X
+SQL ORC MR 3078 3122 62 3.4 293.6 2.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 4818 4824 8 2.2 459.5 1.0X
+SQL Json 4853 4878 35 2.2 462.8 1.0X
+SQL Parquet Vectorized: DataPageV1 255 264 6 41.1 24.3 18.9X
+SQL Parquet Vectorized: DataPageV2 711 716 4 14.7 67.8 6.8X
Review comment:
After [406d176](https://github.com/apache/spark/pull/35262/commits/406d1768019eed1516925847a59bd2ad5fc7883b), should we update bench result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r791393959
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,6 +96,12 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
HadoopInputFile.fromPath(file, configuration), options);
this.reader = new ParquetRowGroupReaderImpl(fileReader);
this.fileSchema = fileReader.getFileMetaData().getSchema();
+ try {
+ this.writerVersion = VersionParser.parse(fileReader.getFileMetaData().getCreatedBy());
+ } catch (Exception e) {
Review comment:
Will other types of exceptions be thrown here, except `VersionParseException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1020778001
@parthchandra Can you update the benchmark running with Java 8 again? The marked data is much slower than before. I'm not sure if it's a noise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1020778001
@parthchandra Can you update the benchmark running with Java 8 again? The marked data is much slower than before. I'm not sure whether this data is reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1020778001
@parthchandra Can you update the benchmark running with Java 8 again? The marked data is much slower than before. I'm not sure if it's a noise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r797417782
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ public void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
+ int length = prefixLength + suffix.length;
+
+ // NOTE: due to PARQUET-246, it is important that we
+ // respect prefixLength which was read from prefixLengthReader,
+ // even for the *first* value of a page. Even though the first
+ // value of the page should have an empty prefix, it may not
+ // because of PARQUET-246.
+
+ // We have to do this to materialize the output
+ if (prefixLength != 0) {
+ // We could do
+ // c.putByteArray(rowId + i, previous, 0, prefixLength);
+ // c.putByteArray(rowId+i, suffix, prefixLength, suffix.length);
+ // previous = c.getBinary(rowId+1);
+ // but it incurs the same cost of copying the values twice _and_ c.getBinary
+ // is a _slow_ byte by byte copy
+ // The following always uses the faster system arraycopy method
+ byte[] out = new byte[length];
+ System.arraycopy(previous, 0, out, 0, prefixLength);
+ System.arraycopy(suffix, 0, out, prefixLength, suffix.length);
+ previous = out;
} else {
- byte[] bytes = new byte[binary.length()];
- buffer.get(bytes);
- c.putByteArray(rowId + i, bytes);
+ previous = suffix;
}
+ outputWriter.write(c, rowId + i, ByteBuffer.wrap(previous), previous.length);
Review comment:
Is line106 always equivalent to ` c.putByteArray(rowId, previous, 0, length)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1071140155
@sunchao, @LuciferYang, @sadikovi thank you for your reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1066760773
@sunchao should we continue this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r810218228
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 14859 14914 77 1.1 944.7 1.0X
+SQL Json 9418 9457 55 1.7 598.8 1.6X
+SQL Parquet Vectorized: DataPageV1 109 128 14 144.6 6.9 136.6X
+SQL Parquet Vectorized: DataPageV2 79 89 8 199.3 5.0 188.3X
+SQL Parquet MR: DataPageV1 1699 1743 62 9.3 108.0 8.7X
+SQL Parquet MR: DataPageV2 1462 1489 38 10.8 93.0 10.2X
+SQL ORC Vectorized 165 200 33 95.3 10.5 90.0X
+SQL ORC MR 1409 1420 16 11.2 89.6 10.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 101 104 3 155.2 6.4 1.0X
+ParquetReader Vectorized: DataPageV2 82 85 5 192.0 5.2 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 48 50 2 324.6 3.1 2.1X
+ParquetReader Vectorized -> Row: DataPageV2 29 31 3 539.4 1.9 3.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 17619 17639 28 0.9 1120.2 1.0X
+SQL Json 10590 10606 23 1.5 673.3 1.7X
+SQL Parquet Vectorized: DataPageV1 178 194 10 88.2 11.3 98.8X
+SQL Parquet Vectorized: DataPageV2 178 188 9 88.2 11.3 98.7X
+SQL Parquet MR: DataPageV1 1884 1887 4 8.4 119.8 9.4X
+SQL Parquet MR: DataPageV2 1689 1742 75 9.3 107.4 10.4X
+SQL ORC Vectorized 162 193 24 97.0 10.3 108.7X
+SQL ORC MR 1505 1552 67 10.5 95.7 11.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 230 236 13 68.3 14.6 1.0X
+ParquetReader Vectorized: DataPageV2 228 233 8 69.1 14.5 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 138 150 26 113.7 8.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 137 140 2 114.5 8.7 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 18488 18494 8 0.9 1175.5 1.0X
+SQL Json 11190 11195 7 1.4 711.4 1.7X
+SQL Parquet Vectorized: DataPageV1 125 155 34 125.7 8.0 147.7X
+SQL Parquet Vectorized: DataPageV2 183 192 9 86.1 11.6 101.2X
+SQL Parquet MR: DataPageV1 2153 2160 10 7.3 136.9 8.6X
+SQL Parquet MR: DataPageV2 1876 1889 18 8.4 119.3 9.9X
+SQL ORC Vectorized 212 257 23 74.4 13.4 87.4X
+SQL ORC MR 1653 1658 7 9.5 105.1 11.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 198 201 5 79.6 12.6 1.0X
+ParquetReader Vectorized: DataPageV2 256 260 3 61.5 16.3 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 193 226 14 81.4 12.3 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 250 253 2 62.8 15.9 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 18614 18703 125 0.8 1183.5 1.0X
+SQL Json 11673 11711 53 1.3 742.2 1.6X
+SQL Parquet Vectorized: DataPageV1 128 154 26 123.1 8.1 145.7X
+SQL Parquet Vectorized: DataPageV2 270 302 23 58.3 17.1 69.0X
+SQL Parquet MR: DataPageV1 2117 2145 39 7.4 134.6 8.8X
+SQL Parquet MR: DataPageV2 1855 1860 7 8.5 117.9 10.0X
+SQL ORC Vectorized 277 292 16 56.7 17.6 67.2X
+SQL ORC MR 1623 1629 9 9.7 103.2 11.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 225 226 1 69.9 14.3 1.0X
+ParquetReader Vectorized: DataPageV2 362 365 2 43.4 23.0 0.6X
+ParquetReader Vectorized -> Row: DataPageV1 193 218 18 81.5 12.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV2 360 366 6 43.7 22.9 0.6X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 23939 23953 20 0.7 1522.0 1.0X
+SQL Json 14445 14449 5 1.1 918.4 1.7X
+SQL Parquet Vectorized: DataPageV1 186 229 28 84.7 11.8 128.9X
+SQL Parquet Vectorized: DataPageV2 459 493 25 34.3 29.2 52.2X
+SQL Parquet MR: DataPageV1 2180 2184 7 7.2 138.6 11.0X
+SQL Parquet MR: DataPageV2 1954 1973 27 8.1 124.2 12.3X
+SQL ORC Vectorized 368 392 24 42.8 23.4 65.1X
+SQL ORC MR 1793 1794 2 8.8 114.0 13.4X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 280 293 18 56.1 17.8 1.0X
+ParquetReader Vectorized: DataPageV2 577 602 48 27.3 36.7 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 314 321 10 50.1 19.9 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 581 584 4 27.1 37.0 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 19278 19291 18 0.8 1225.6 1.0X
+SQL Json 13366 13381 21 1.2 849.8 1.4X
+SQL Parquet Vectorized: DataPageV1 130 152 23 120.8 8.3 148.1X
+SQL Parquet Vectorized: DataPageV2 135 157 17 116.8 8.6 143.2X
+SQL Parquet MR: DataPageV1 2126 2137 15 7.4 135.2 9.1X
+SQL Parquet MR: DataPageV2 1970 1985 21 8.0 125.2 9.8X
+SQL ORC Vectorized 387 396 11 40.7 24.6 49.8X
+SQL ORC MR 1831 1832 1 8.6 116.4 10.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 194 197 5 81.1 12.3 1.0X
+ParquetReader Vectorized: DataPageV2 194 197 7 81.2 12.3 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 225 253 18 69.9 14.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 224 252 18 70.2 14.2 0.9X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 24691 24705 19 0.6 1569.8 1.0X
+SQL Json 18028 18028 0 0.9 1146.2 1.4X
+SQL Parquet Vectorized: DataPageV1 190 225 28 83.0 12.0 130.3X
+SQL Parquet Vectorized: DataPageV2 188 230 26 83.9 11.9 131.7X
+SQL Parquet MR: DataPageV1 2362 2365 4 6.7 150.2 10.5X
+SQL Parquet MR: DataPageV2 2061 2078 25 7.6 131.0 12.0X
+SQL ORC Vectorized 499 524 37 31.6 31.7 49.5X
+SQL ORC MR 1870 1880 14 8.4 118.9 13.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 276 295 21 57.0 17.5 1.0X
+ParquetReader Vectorized: DataPageV2 278 289 17 56.6 17.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 315 326 15 50.0 20.0 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 315 323 8 49.9 20.0 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 16840 16908 96 0.6 1606.0 1.0X
+SQL Json 12496 12513 25 0.8 1191.7 1.3X
+SQL Parquet Vectorized: DataPageV1 2169 2172 5 4.8 206.9 7.8X
+SQL Parquet Vectorized: DataPageV2 3102 3119 24 3.4 295.9 5.4X
+SQL Parquet MR: DataPageV1 4140 4144 5 2.5 394.8 4.1X
+SQL Parquet MR: DataPageV2 3988 3996 12 2.6 380.3 4.2X
+SQL ORC Vectorized 2180 2196 23 4.8 207.9 7.7X
+SQL ORC MR 3765 3766 2 2.8 359.0 4.5X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 9960 9960 0 1.1 949.8 1.0X
+SQL Json 7625 7712 123 1.4 727.2 1.3X
+SQL Parquet Vectorized: DataPageV1 577 582 6 18.2 55.0 17.3X
+SQL Parquet Vectorized: DataPageV2 584 592 6 18.0 55.7 17.1X
+SQL Parquet MR: DataPageV1 1722 1736 19 6.1 164.2 5.8X
+SQL Parquet MR: DataPageV2 1662 1668 9 6.3 158.5 6.0X
+SQL ORC Vectorized 483 524 27 21.7 46.1 20.6X
+SQL ORC MR 1841 1850 14 5.7 175.5 5.4X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 23787 23788 2 0.7 1512.3 1.0X
+Data column - Json 13993 14011 25 1.1 889.7 1.7X
+Data column - Parquet Vectorized: DataPageV1 184 235 36 85.4 11.7 129.2X
+Data column - Parquet Vectorized: DataPageV2 531 542 15 29.6 33.7 44.8X
+Data column - Parquet MR: DataPageV1 2539 2547 13 6.2 161.4 9.4X
+Data column - Parquet MR: DataPageV2 2299 2301 3 6.8 146.2 10.3X
+Data column - ORC Vectorized 379 403 23 41.5 24.1 62.8X
+Data column - ORC MR 2047 2070 33 7.7 130.1 11.6X
+Partition column - CSV 6834 6835 1 2.3 434.5 3.5X
+Partition column - Json 11444 11478 49 1.4 727.6 2.1X
+Partition column - Parquet Vectorized: DataPageV1 51 71 22 308.6 3.2 466.7X
+Partition column - Parquet Vectorized: DataPageV2 51 61 16 310.5 3.2 469.5X
+Partition column - Parquet MR: DataPageV1 1203 1214 15 13.1 76.5 19.8X
+Partition column - Parquet MR: DataPageV2 1210 1224 20 13.0 76.9 19.7X
+Partition column - ORC Vectorized 52 67 14 303.1 3.3 458.4X
+Partition column - ORC MR 1338 1342 5 11.8 85.1 17.8X
+Both columns - CSV 24051 24052 2 0.7 1529.1 1.0X
+Both columns - Json 15016 15030 20 1.0 954.7 1.6X
+Both columns - Parquet Vectorized: DataPageV1 235 269 27 66.9 15.0 101.2X
+Both columns - Parquet Vectorized: DataPageV2 563 617 60 27.9 35.8 42.2X
+Both columns - Parquet MR: DataPageV1 2525 2555 43 6.2 160.5 9.4X
+Both columns - Parquet MR: DataPageV2 2256 2267 15 7.0 143.5 10.5X
+Both columns - ORC Vectorized 407 454 51 38.7 25.9 58.5X
+Both columns - ORC MR 2153 2155 2 7.3 136.9 11.0X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 11812 11849 53 0.9 1126.4 1.0X
+SQL Json 11454 11467 18 0.9 1092.3 1.0X
+SQL Parquet Vectorized: DataPageV1 1250 1276 37 8.4 119.2 9.5X
+SQL Parquet Vectorized: DataPageV2 2248 2261 17 4.7 214.4 5.3X
+SQL Parquet MR: DataPageV1 3629 3630 1 2.9 346.1 3.3X
+SQL Parquet MR: DataPageV2 3929 3934 6 2.7 374.7 3.0X
+ParquetReader Vectorized: DataPageV1 921 922 2 11.4 87.8 12.8X
+ParquetReader Vectorized: DataPageV2 1890 1890 0 5.5 180.3 6.2X
+SQL ORC Vectorized 1079 1105 36 9.7 102.9 10.9X
+SQL ORC MR 3042 3070 40 3.4 290.1 3.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 8399 8410 16 1.2 801.0 1.0X
+SQL Json 8892 8905 18 1.2 848.0 0.9X
+SQL Parquet Vectorized: DataPageV1 1065 1092 38 9.8 101.6 7.9X
+SQL Parquet Vectorized: DataPageV2 1747 1747 0 6.0 166.6 4.8X
+SQL Parquet MR: DataPageV1 2718 2719 1 3.9 259.2 3.1X
+SQL Parquet MR: DataPageV2 2955 2964 12 3.5 281.8 2.8X
+ParquetReader Vectorized: DataPageV1 1082 1084 3 9.7 103.2 7.8X
+ParquetReader Vectorized: DataPageV2 1707 1713 9 6.1 162.8 4.9X
+SQL ORC Vectorized 1345 1357 17 7.8 128.3 6.2X
+SQL ORC MR 3012 3046 47 3.5 287.3 2.8X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 6169 6176 10 1.7 588.3 1.0X
+SQL Json 5352 5376 35 2.0 510.4 1.2X
+SQL Parquet Vectorized: DataPageV1 248 255 7 42.3 23.6 24.9X
+SQL Parquet Vectorized: DataPageV2 364 372 10 28.8 34.7 17.0X
+SQL Parquet MR: DataPageV1 1624 1626 3 6.5 154.9 3.8X
+SQL Parquet MR: DataPageV2 1520 1526 8 6.9 145.0 4.1X
+ParquetReader Vectorized: DataPageV1 259 262 1 40.4 24.7 23.8X
+ParquetReader Vectorized: DataPageV2 376 378 2 27.9 35.9 16.4X
Review comment:
Hmm this still shows a degradation (69% vs 80% originally). I wonder why vectorized reader doesn't help here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-36879][SQL][FOLLOWUP] Support Parquet v2 data page encodings for the vectorized path
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1017816296
@sunchao, @dongjoon-hyun, @viirya, @LuciferYang could you please review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #35262: [SPARK-36879][SQL][FOLLOWUP] Support Parquet v2 data page encodings for the vectorized path
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1017821652
Yea, this looks more than a follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r791326157
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
Review comment:
this `valueCount` can also be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1021817183
> Updated the JDK 8 benchmark results as well.
After comparing the new bench data, I find that the data corresponding to `Parquet Data Page V2` in the two test cases `String with Nulls Scan (50.0%) ` and `String with Nulls Scan (95.0%)` is slower than the previous pr :
| | String with Nulls Scan (50.0%) | String with Nulls Scan (95.0%) |
| :--- | :--- |:--- |
| before | 145.7 ns/per row | 25.2 ns/per row |
| after | 228.7 ns/per row | 77.9 ns/per row |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r791231334
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -147,12 +147,19 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
withTempPath { dir =>
val path = s"${dir.getCanonicalPath}/test.parquet"
- val data = (1 to 3).map { i =>
- ( i, i.toLong, i.toShort, Array[Byte](i.toByte), s"test_${i}",
- DateTimeUtils.fromJavaDate(Date.valueOf(s"2021-11-0" + i)),
- DateTimeUtils.fromJavaTimestamp(Timestamp.valueOf(s"2020-11-01 12:00:0" + i)),
- Period.of(1, i, 0), Duration.ofMillis(i * 100),
- new BigDecimal(java.lang.Long.toUnsignedString(i*100000))
+ // Have more than 2 * 4096 records (so we have multiple tasks and each task
+ // reads at least twice from the reader). This will catch any issues with state
+ // maintained by the reader(s)
+ // Add at least one string with a null
+ val data = (1 to 8197).map { i =>
+ ( i,
+ i.toLong, i.toShort, Array[Byte](i.toByte),
+ if (i % 2 == 1) s"test_${i}" else null,
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ prefixLengthVector = new OffHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OffHeapColumnVector(valueCount, BinaryType);
+ } else {
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ }
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, null, 0,
+ (w, r, v, l) ->
Review comment:
What you say makes sense: that the reference to the external variable will cause multiple object instantiation. Thank you for doing this reasearch!
I tried something similar but with the unit test in ParquetEncodingSuite and see only a single instance of the lambda created (not sure why).
I've changed the code to use a WritableVector of size 1 which eliminates the need to access the variable directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1022462107
> > Updated the JDK 8 benchmark results as well.
>
> After comparing the new bench data, I find that the data corresponding to `Parquet Data Page V2` in the two test cases `String with Nulls Scan (50.0%) ` and `String with Nulls Scan (95.0%)` is relatively slower than the previous pr (although the CPU frequency of the testing machine is reduced):
>
> before after
> String with Nulls Scan (50.0%) 145.7 ns/per row 228.7 ns/per row
> String with Nulls Scan (95.0%) 25.2 ns/per row 77.9 ns/per row
It's hard to reasonably compare the numbers across runs (even though the difference is substantial) because of the difference in the environment. Let me do a profile run to see if any obvious bottlenecks stand out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r793451912
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
Review comment:
Maybe we should call `lengthsVector.putInts(0, valueCount, 0);` to ensure this init value of OffHeapColumnVector, or use other ways to avoid reading unexpected values in `line 75` when `memoryMode is MemoryMode.OFF_HEAP`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1021817183
> Updated the JDK 8 benchmark results as well.
After comparing the new bench data, I find that the data corresponding to `Parquet Data Page V2` in the two test cases `String with Nulls Scan (50.0%) ` and `String with Nulls Scan (95.0%)` is slower than the previous pr :
| | before | after |
| :--- | :--- |:--- |
| String with Nulls Scan (50.0%) | 145.7 ns/per row | 228.7 ns/per row |
| String with Nulls Scan (95.0%) | 25.2 ns/per row | 77.9 ns/per row |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r797417782
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ public void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
+ int length = prefixLength + suffix.length;
+
+ // NOTE: due to PARQUET-246, it is important that we
+ // respect prefixLength which was read from prefixLengthReader,
+ // even for the *first* value of a page. Even though the first
+ // value of the page should have an empty prefix, it may not
+ // because of PARQUET-246.
+
+ // We have to do this to materialize the output
+ if (prefixLength != 0) {
+ // We could do
+ // c.putByteArray(rowId + i, previous, 0, prefixLength);
+ // c.putByteArray(rowId+i, suffix, prefixLength, suffix.length);
+ // previous = c.getBinary(rowId+1);
+ // but it incurs the same cost of copying the values twice _and_ c.getBinary
+ // is a _slow_ byte by byte copy
+ // The following always uses the faster system arraycopy method
+ byte[] out = new byte[length];
+ System.arraycopy(previous, 0, out, 0, prefixLength);
+ System.arraycopy(suffix, 0, out, prefixLength, suffix.length);
+ previous = out;
} else {
- byte[] bytes = new byte[binary.length()];
- buffer.get(bytes);
- c.putByteArray(rowId + i, bytes);
+ previous = suffix;
}
+ outputWriter.write(c, rowId + i, ByteBuffer.wrap(previous), previous.length);
Review comment:
Is line106 always equivalent to ` c.putByteArray(rowId, previous, 0, length)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794225009
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Right, I missed that @LuciferYang was talking about null values. This method gets called only for non-null values.
I meant that if the ith value is an empty string and we simply do not write the ith value to the vector, does the vector guarantee that the ith value will be treated as an empty string?
If so we can simply not write the value if the length is zero. We'll save a couple of function calls for empty strings but we will add an if condition for every value. Probably not worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1023545638
Thank you for finding this issue! Let me address this and add the unit test(s) as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1023832057
@LuciferYang you were right, the vector needed initialization.
Also added a memory mode parameter for the unit test. I increased the size of the data generated to reproduce the issue and then applied the fix. Also ran the benchmark with the off heap enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1024049100
I have another question: when `memoryMode` is `MemoryMode.OFF_HEAP`, some `OffHeapColumnVector`s will be create like `binaryValVector`, `lengthsVector` and so on. `OffHeapColumnVector` needs to call the `close` method manually to release the allocated memory, but I can't find the corresponding call.
I think there may be a memory leak. @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1025369374
> Oh dear. This will take some fixing. Thank you for catching this!
I think we can also remove the `OFF_HEAP` mode support in current pr, (Sorry, I haven't seen the advantages of `OFF_HEAP` mode in the current scenario and other readers haven't used `OFF_HEAP` mode yet) then add the unified resource release mechanism and `OFF_HEAP` mode support in another pr. This may help to make the current pr merge faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1061141261
@sunchao IO merged your changes into the PR. Also updated the benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r802888652
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,7 +180,7 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
readState.resetForNewPage(pageValueCount, pageFirstRowIndex);
}
PrimitiveType.PrimitiveTypeName typeName =
- descriptor.getPrimitiveType().getPrimitiveTypeName();
+ descriptor.getPrimitiveType().getPrimitiveTypeName();
Review comment:
Wierd. Shows up as changed in my IDE, but not by me ! Undid the change anyway.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -295,13 +307,13 @@ private ValuesReader getValuesReader(Encoding encoding) {
return new VectorizedDeltaBinaryPackedReader();
case RLE:
PrimitiveType.PrimitiveTypeName typeName =
- this.descriptor.getPrimitiveType().getPrimitiveTypeName();
+ this.descriptor.getPrimitiveType().getPrimitiveTypeName();
// RLE encoding only supports boolean type `Values`, and `bitwidth` is always 1.
if (typeName == BOOLEAN) {
return new VectorizedRleValuesReader(1);
} else {
throw new UnsupportedOperationException(
- "RLE encoding is not supported for values of type: " + typeName);
+ "RLE encoding is not supported for values of type: " + typeName);
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
Review comment:
The way this get's called, the valueCount is pretty much going to be for a run of non-null values to be read. So it doesn't matter. But, it is more correct to use totalValueCount I guess. And it is consistent with the previous line (#60)
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
Review comment:
I had this there because the initial PR for DeltaBinary also had this check but had not validated the contract. I changed this and added a Precondition check and one unit test that interleaves read and skip failed. This test is from the Parquet implementation and so it appears that Parquet's specification is that we could call the API with total == 0.
I could still remove the check and update the unit test if you think that we should.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
Review comment:
Yes we should initialize it. Later, in `readValues` we copy this into the output without checking if it is null (and we want to avoid having to check that).
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
+ int length = prefixLength + suffix.length;
+
+ // NOTE: due to PARQUET-246, it is important that we
+ // respect prefixLength which was read from prefixLengthReader,
+ // even for the *first* value of a page. Even though the first
+ // value of the page should have an empty prefix, it may not
+ // because of PARQUET-246.
+
+ // We have to do this to materialize the output
+ if (prefixLength != 0) {
+ // We could do
+ // c.putByteArray(rowId + i, previous, 0, prefixLength);
+ // c.putByteArray(rowId+i, suffix, prefixLength, suffix.length);
+ // previous = c.getBinary(rowId+1);
+ // but it incurs the same cost of copying the values twice _and_ c.getBinary
+ // is a _slow_ byte by byte copy
+ // The following always uses the faster system arraycopy method
+ byte[] out = new byte[length];
+ System.arraycopy(previous, 0, out, 0, prefixLength);
+ System.arraycopy(suffix, 0, out, prefixLength, suffix.length);
+ previous = out;
} else {
- byte[] bytes = new byte[binary.length()];
- buffer.get(bytes);
- c.putByteArray(rowId + i, bytes);
+ previous = suffix;
}
+ outputWriter.write(c, rowId + i, ByteBuffer.wrap(previous), previous.length);
+ currentRow++;
}
}
@Override
- public void skipBinary(int total) {
- for (int i = 0; i < total; i++) {
- deltaByteArrayReader.skip();
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ readValues(total, c, rowId, ByteBufferOutputWriter::writeArrayByteBuffer);
+ }
+
+ /**
+ * There was a bug (PARQUET-246) in which DeltaByteArrayWriter's reset() method did not clear the
+ * previous value state that it tracks internally. This resulted in the first value of all pages
Review comment:
Last value of the previous page (except for the first page where it is zero).
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
This is how it was initially. @LuciferYang, would you mind if I removed the check?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -295,13 +307,13 @@ private ValuesReader getValuesReader(Encoding encoding) {
return new VectorizedDeltaBinaryPackedReader();
case RLE:
PrimitiveType.PrimitiveTypeName typeName =
- this.descriptor.getPrimitiveType().getPrimitiveTypeName();
+ this.descriptor.getPrimitiveType().getPrimitiveTypeName();
Review comment:
Same as before. Changed it anyway.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
+ int length = prefixLength + suffix.length;
+
+ // NOTE: due to PARQUET-246, it is important that we
Review comment:
Moved the comment to an earlier location in the block.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
Review comment:
Of course
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader() {
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
+ if (length > 0) {
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
+ }
+ }
+ currentRow += total;
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ if (total == 0) {
+ return;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(currentRow + i);
+ int remaining = length;
+ while (remaining > 0) {
+ remaining -= in.skip(length);
Review comment:
So it should. Corrected.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
Review comment:
The prefix lengths have already been decoded in the constructor and read into the prefixLengthVector. This is just reading the value from the vector.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
Review comment:
Sure
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
Review comment:
Right, that's what I had to start with. I could have made the change to handle the memory leak in this PR, but probably safer to do in a new PR. ( Followup JIRA - https://issues.apache.org/jira/browse/SPARK-38169)
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -69,6 +71,7 @@
protected MessageType fileSchema;
protected MessageType requestedSchema;
protected StructType sparkSchema;
+ protected ParsedVersion writerVersion;
Review comment:
done
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 10153 10161 12 1.5 645.5 1.0X
+SQL Json 8463 8512 69 1.9 538.0 1.2X
+SQL Parquet Vectorized: DataPageV1 131 149 14 120.0 8.3 77.5X
+SQL Parquet Vectorized: DataPageV2 98 112 15 161.2 6.2 104.0X
+SQL Parquet MR: DataPageV1 1968 1968 0 8.0 125.1 5.2X
+SQL Parquet MR: DataPageV2 1735 1739 6 9.1 110.3 5.9X
+SQL ORC Vectorized 164 198 41 96.0 10.4 62.0X
+SQL ORC MR 1572 1581 12 10.0 100.0 6.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 102 107 14 154.6 6.5 1.0X
+ParquetReader Vectorized: DataPageV2 83 88 10 189.1 5.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 57 59 3 275.7 3.6 1.8X
+ParquetReader Vectorized -> Row: DataPageV2 38 40 3 416.3 2.4 2.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12263 12285 31 1.3 779.6 1.0X
+SQL Json 9495 9501 9 1.7 603.6 1.3X
+SQL Parquet Vectorized: DataPageV1 162 175 10 97.1 10.3 75.7X
+SQL Parquet Vectorized: DataPageV2 161 172 12 97.9 10.2 76.4X
+SQL Parquet MR: DataPageV1 2074 2105 44 7.6 131.9 5.9X
+SQL Parquet MR: DataPageV2 1974 1981 9 8.0 125.5 6.2X
+SQL ORC Vectorized 187 218 30 84.3 11.9 65.7X
+SQL ORC MR 1529 1553 34 10.3 97.2 8.0X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 205 214 12 76.8 13.0 1.0X
+ParquetReader Vectorized: DataPageV2 204 211 10 77.2 13.0 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 122 132 24 128.7 7.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 122 126 6 128.4 7.8 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12946 12955 12 1.2 823.1 1.0X
+SQL Json 9871 9876 8 1.6 627.6 1.3X
+SQL Parquet Vectorized: DataPageV1 157 200 34 100.0 10.0 82.3X
+SQL Parquet Vectorized: DataPageV2 229 242 14 68.8 14.5 56.7X
+SQL Parquet MR: DataPageV1 2388 2389 2 6.6 151.8 5.4X
+SQL Parquet MR: DataPageV2 2080 2087 10 7.6 132.2 6.2X
+SQL ORC Vectorized 240 285 23 65.6 15.2 54.0X
+SQL ORC MR 1699 1732 46 9.3 108.0 7.6X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 242 245 5 65.0 15.4 1.0X
+ParquetReader Vectorized: DataPageV2 309 314 9 50.9 19.7 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 227 268 18 69.3 14.4 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 294 312 25 53.5 18.7 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14331 14347 22 1.1 911.2 1.0X
+SQL Json 10406 10434 40 1.5 661.6 1.4X
+SQL Parquet Vectorized: DataPageV1 153 196 41 102.7 9.7 93.6X
+SQL Parquet Vectorized: DataPageV2 378 415 30 41.6 24.0 37.9X
+SQL Parquet MR: DataPageV1 2439 2446 11 6.4 155.1 5.9X
+SQL Parquet MR: DataPageV2 2181 2188 10 7.2 138.7 6.6X
+SQL ORC Vectorized 320 346 25 49.2 20.3 44.8X
+SQL ORC MR 1851 1853 3 8.5 117.7 7.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 258 262 9 60.9 16.4 1.0X
+ParquetReader Vectorized: DataPageV2 481 484 3 32.7 30.6 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 250 275 26 62.9 15.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 475 502 27 33.1 30.2 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 18450 18451 2 0.9 1173.0 1.0X
+SQL Json 12553 12562 13 1.3 798.1 1.5X
+SQL Parquet Vectorized: DataPageV1 259 272 12 60.8 16.5 71.3X
+SQL Parquet Vectorized: DataPageV2 534 566 22 29.4 34.0 34.5X
+SQL Parquet MR: DataPageV1 2529 2537 11 6.2 160.8 7.3X
+SQL Parquet MR: DataPageV2 2331 2334 4 6.7 148.2 7.9X
+SQL ORC Vectorized 424 460 36 37.1 27.0 43.5X
+SQL ORC MR 2009 2023 20 7.8 127.7 9.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 322 338 21 48.9 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 674 683 12 23.3 42.9 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 352 358 9 44.7 22.4 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 628 660 22 25.0 39.9 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14806 14816 14 1.1 941.3 1.0X
+SQL Json 11968 11969 1 1.3 760.9 1.2X
+SQL Parquet Vectorized: DataPageV1 150 184 26 105.0 9.5 98.8X
+SQL Parquet Vectorized: DataPageV2 147 183 32 107.2 9.3 100.9X
+SQL Parquet MR: DataPageV1 2338 2352 19 6.7 148.7 6.3X
+SQL Parquet MR: DataPageV2 2221 2267 65 7.1 141.2 6.7X
+SQL ORC Vectorized 475 494 29 33.1 30.2 31.1X
+SQL ORC MR 1967 1978 16 8.0 125.1 7.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 236 241 8 66.7 15.0 1.0X
+ParquetReader Vectorized: DataPageV2 237 241 9 66.3 15.1 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 218 244 25 72.1 13.9 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 218 251 21 72.2 13.8 1.1X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 19335 19346 15 0.8 1229.3 1.0X
+SQL Json 16112 16121 13 1.0 1024.4 1.2X
+SQL Parquet Vectorized: DataPageV1 257 278 29 61.1 16.4 75.1X
+SQL Parquet Vectorized: DataPageV2 258 268 9 60.9 16.4 74.9X
+SQL Parquet MR: DataPageV1 2542 2557 20 6.2 161.6 7.6X
+SQL Parquet MR: DataPageV2 2416 2439 32 6.5 153.6 8.0X
+SQL ORC Vectorized 593 605 18 26.5 37.7 32.6X
+SQL ORC MR 2134 2141 11 7.4 135.7 9.1X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 322 346 24 48.8 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 326 326 1 48.3 20.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 350 359 9 44.9 22.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 348 358 10 45.2 22.1 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 13899 14000 142 0.8 1325.5 1.0X
+SQL Json 11275 11289 20 0.9 1075.3 1.2X
+SQL Parquet Vectorized: DataPageV1 2092 2107 21 5.0 199.5 6.6X
+SQL Parquet Vectorized: DataPageV2 3073 3074 2 3.4 293.0 4.5X
+SQL Parquet MR: DataPageV1 4192 4212 29 2.5 399.8 3.3X
+SQL Parquet MR: DataPageV2 4133 4194 87 2.5 394.1 3.4X
+SQL ORC Vectorized 2218 2219 1 4.7 211.5 6.3X
+SQL ORC MR 3767 3776 12 2.8 359.3 3.7X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 7367 7387 28 1.4 702.6 1.0X
+SQL Json 6817 6817 0 1.5 650.1 1.1X
+SQL Parquet Vectorized: DataPageV1 602 618 15 17.4 57.5 12.2X
+SQL Parquet Vectorized: DataPageV2 599 610 15 17.5 57.1 12.3X
+SQL Parquet MR: DataPageV1 1888 1936 68 5.6 180.0 3.9X
+SQL Parquet MR: DataPageV2 2000 2018 25 5.2 190.7 3.7X
+SQL ORC Vectorized 527 545 22 19.9 50.2 14.0X
+SQL ORC MR 1916 1927 16 5.5 182.7 3.8X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 19239 19257 25 0.8 1223.2 1.0X
+Data column - Json 12387 12393 8 1.3 787.6 1.6X
+Data column - Parquet Vectorized: DataPageV1 227 269 25 69.2 14.5 84.6X
+Data column - Parquet Vectorized: DataPageV2 612 651 28 25.7 38.9 31.4X
+Data column - Parquet MR: DataPageV1 2989 3016 39 5.3 190.0 6.4X
+Data column - Parquet MR: DataPageV2 2750 2754 5 5.7 174.8 7.0X
+Data column - ORC Vectorized 426 467 33 37.0 27.1 45.2X
+Data column - ORC MR 2513 2538 35 6.3 159.8 7.7X
+Partition column - CSV 6623 6627 5 2.4 421.1 2.9X
+Partition column - Json 10234 10235 2 1.5 650.7 1.9X
+Partition column - Parquet Vectorized: DataPageV1 56 73 19 279.8 3.6 342.2X
+Partition column - Parquet Vectorized: DataPageV2 57 72 19 278.3 3.6 340.4X
+Partition column - Parquet MR: DataPageV1 1392 1417 36 11.3 88.5 13.8X
+Partition column - Parquet MR: DataPageV2 1390 1416 37 11.3 88.4 13.8X
+Partition column - ORC Vectorized 56 89 36 283.2 3.5 346.4X
+Partition column - ORC MR 1578 1581 4 10.0 100.4 12.2X
+Both columns - CSV 19178 19181 4 0.8 1219.3 1.0X
+Both columns - Json 13104 13105 1 1.2 833.1 1.5X
+Both columns - Parquet Vectorized: DataPageV1 314 338 21 50.2 19.9 61.4X
+Both columns - Parquet Vectorized: DataPageV2 708 741 54 22.2 45.0 27.2X
+Both columns - Parquet MR: DataPageV1 3083 3105 31 5.1 196.0 6.2X
+Both columns - Parquet MR: DataPageV2 2897 2901 6 5.4 184.2 6.6X
+Both columns - ORC Vectorized 456 504 39 34.5 29.0 42.1X
+Both columns - ORC MR 2594 2597 4 6.1 164.9 7.4X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 9165 9257 130 1.1 874.1 1.0X
+SQL Json 10230 10234 7 1.0 975.6 0.9X
+SQL Parquet Vectorized: DataPageV1 1275 1315 56 8.2 121.6 7.2X
+SQL Parquet Vectorized: DataPageV2 2406 2407 0 4.4 229.5 3.8X
+SQL Parquet MR: DataPageV1 4005 4009 7 2.6 381.9 2.3X
+SQL Parquet MR: DataPageV2 4358 4366 12 2.4 415.6 2.1X
+ParquetReader Vectorized: DataPageV1 985 995 13 10.6 94.0 9.3X
+ParquetReader Vectorized: DataPageV2 2039 2061 32 5.1 194.4 4.5X
+SQL ORC Vectorized 1048 1072 34 10.0 99.9 8.7X
+SQL ORC MR 3179 3196 24 3.3 303.2 2.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 6749 6750 2 1.6 643.6 1.0X
+SQL Json 7960 7967 10 1.3 759.1 0.8X
+SQL Parquet Vectorized: DataPageV1 1078 1105 37 9.7 102.8 6.3X
+SQL Parquet Vectorized: DataPageV2 1939 1941 3 5.4 184.9 3.5X
+SQL Parquet MR: DataPageV1 3090 3099 13 3.4 294.7 2.2X
+SQL Parquet MR: DataPageV2 3274 3286 17 3.2 312.3 2.1X
+ParquetReader Vectorized: DataPageV1 1058 1067 13 9.9 100.9 6.4X
+ParquetReader Vectorized: DataPageV2 1847 1848 2 5.7 176.2 3.7X
+SQL ORC Vectorized 1307 1307 0 8.0 124.6 5.2X
+SQL ORC MR 3078 3122 62 3.4 293.6 2.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 4818 4824 8 2.2 459.5 1.0X
+SQL Json 4853 4878 35 2.2 462.8 1.0X
+SQL Parquet Vectorized: DataPageV1 255 264 6 41.1 24.3 18.9X
+SQL Parquet Vectorized: DataPageV2 711 716 4 14.7 67.8 6.8X
Review comment:
I haven't quite figured out how to run the benchmark in a profiler so I can see the bottlenecks. I did run the unit test in a profiler but the decoder was not anywhere in the picture of where time was being spent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r804254672
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ private void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
Review comment:
I think it's better to remove it, for performance reason. The vectorized reader in Spark should always call this method with `total > 0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r790403317
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ prefixLengthVector = new OffHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OffHeapColumnVector(valueCount, BinaryType);
+ } else {
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ }
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, null, 0,
+ (w, r, v, l) ->
Review comment:
cc @rednaxelafx , can you help check whether multiple objects or one object will be generated In this lambda scene?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1020778001
@parthchandra Can you update the benchmark running with Java 8 again? The marked data is much slower than before. I'm not sure whether this data is reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r797418998
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ public void readValues(int total, WritableColumnVector c, int rowId,
+ ByteBufferOutputWriter outputWriter) {
+ if (total == 0) {
+ return;
+ }
+
for (int i = 0; i < total; i++) {
- Binary binary = deltaByteArrayReader.readBytes();
- ByteBuffer buffer = binary.toByteBuffer();
- if (buffer.hasArray()) {
- c.putByteArray(rowId + i, buffer.array(), buffer.arrayOffset() + buffer.position(),
- binary.length());
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ byte[] suffix = suffixVector.getBinary(currentRow);
+ // This does not copy bytes
+ int length = prefixLength + suffix.length;
+
+ // NOTE: due to PARQUET-246, it is important that we
+ // respect prefixLength which was read from prefixLengthReader,
+ // even for the *first* value of a page. Even though the first
+ // value of the page should have an empty prefix, it may not
+ // because of PARQUET-246.
+
+ // We have to do this to materialize the output
+ if (prefixLength != 0) {
+ // We could do
+ // c.putByteArray(rowId + i, previous, 0, prefixLength);
+ // c.putByteArray(rowId+i, suffix, prefixLength, suffix.length);
+ // previous = c.getBinary(rowId+1);
+ // but it incurs the same cost of copying the values twice _and_ c.getBinary
+ // is a _slow_ byte by byte copy
+ // The following always uses the faster system arraycopy method
+ byte[] out = new byte[length];
+ System.arraycopy(previous, 0, out, 0, prefixLength);
+ System.arraycopy(suffix, 0, out, prefixLength, suffix.length);
+ previous = out;
} else {
- byte[] bytes = new byte[binary.length()];
- buffer.get(bytes);
- c.putByteArray(rowId + i, bytes);
+ previous = suffix;
}
+ outputWriter.write(c, rowId + i, ByteBuffer.wrap(previous), previous.length);
+ currentRow++;
Review comment:
should we change line107 to `currentRow += total`and move it out of the loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r798135449
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -66,20 +57,18 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
}
ByteBuffer buffer;
ByteBufferOutputWriter outputWriter;
- if (memoryMode == MemoryMode.OFF_HEAP) {
- outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
- } else {
- outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
- }
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -66,20 +57,18 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
}
ByteBuffer buffer;
ByteBufferOutputWriter outputWriter;
- if (memoryMode == MemoryMode.OFF_HEAP) {
- outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
- } else {
- outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
- }
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
int length;
for (int i = 0; i < total; i++) {
length = lengthsVector.getInt(rowId + i);
- try {
- buffer = in.slice(length);
- } catch (EOFException e) {
- throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ if (length > 0) {
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
}
- outputWriter.write(c, rowId + i, buffer, length);
currentRow++;
Review comment:
Done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ public void readValues(int total, WritableColumnVector c, int rowId,
Review comment:
changed to private
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -66,20 +57,18 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
}
ByteBuffer buffer;
ByteBufferOutputWriter outputWriter;
- if (memoryMode == MemoryMode.OFF_HEAP) {
- outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
- } else {
- outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
- }
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
int length;
for (int i = 0; i < total; i++) {
length = lengthsVector.getInt(rowId + i);
- try {
- buffer = in.slice(length);
- } catch (EOFException e) {
- throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ if (length > 0) {
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
}
- outputWriter.write(c, rowId + i, buffer, length);
currentRow++;
Review comment:
Note that there is a similar line in `VectorizedDeltaByteArrayReader` that cannot be moved out of the loop because the incremented value is accessed in each iteration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r806497889
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Got it, agree with @sunchao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1039934162
Can you re-trigger GA? @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789320438
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
Review comment:
hmm... may be a stupid question, but why does `VectorizedDeltaByteArrayReader` need to hold `valueCount`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r820000445
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,115 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader;
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ // temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.prefixLengthReader = new VectorizedDeltaBinaryPackedReader();
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(prefixLengthReader.getTotalValueCount(), suffixVector, 0);
Review comment:
Instead of eagerly read the suffixes, we can have a method in `VectorizedDeltaLengthByteArrayReader` that just return the suffix at `rowId`:
```java
public ByteBuffer getBytes(int rowId) {
int length = lengthsVector.getInt(rowId);
try {
return in.slice(length);
} catch (EOFException e) {
throw new ParquetDecodingException("Failed to read " + length + " bytes");
}
}
```
I tried this approach [here](https://github.com/sunchao/spark/commit/6d15a86be25d9911f9f7ee8d866afeb170cc7fa4), and it can improve the benchmark.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader;
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader() {
+ lengthReader = new VectorizedDeltaBinaryPackedReader();
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
+ }
+ currentRow += total;
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ if (total == 0) {
Review comment:
We can remove this too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r828482903
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java
##########
@@ -221,6 +221,13 @@ protected UTF8String getBytesAsUTF8String(int rowId, int count) {
return UTF8String.fromAddress(null, data + rowId, count);
}
+ @Override
+ public ByteBuffer getBytesUnsafe(int rowId, int count) {
Review comment:
It seems fine to use `getBytes` here also - the function call will be inlined by JIT if this is in a hot path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1024049100
I have another question:
When `memoryMode` is `MemoryMode.OFF_HEAP`, some `OffHeapColumnVector` will be create like `binaryValVector`, `lengthsVector` and so on. `OffHeapColumnVector` needs to call the `close` method manually to release the allocated memory, but I can't find the corresponding call.
SO there may be a memory leak. @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r793451912
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
Review comment:
Maybe we should call `lengthsVector.putInts(0, valueCount, 0);` to ensure this init value of OffHeapColumnVector, or use other means to avoid reading unexpected values in `line 75`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r826682999
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,50 +16,126 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
Review comment:
Do we have a clear import order definition for static import ? @sunchao @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1069705260
Updated getBytesUnsafe to getByteBuffer and cleaned up the off-heap implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r800948323
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
will take a look soon, sorry missed this ping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r804262218
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 10153 10161 12 1.5 645.5 1.0X
+SQL Json 8463 8512 69 1.9 538.0 1.2X
+SQL Parquet Vectorized: DataPageV1 131 149 14 120.0 8.3 77.5X
+SQL Parquet Vectorized: DataPageV2 98 112 15 161.2 6.2 104.0X
+SQL Parquet MR: DataPageV1 1968 1968 0 8.0 125.1 5.2X
+SQL Parquet MR: DataPageV2 1735 1739 6 9.1 110.3 5.9X
+SQL ORC Vectorized 164 198 41 96.0 10.4 62.0X
+SQL ORC MR 1572 1581 12 10.0 100.0 6.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 102 107 14 154.6 6.5 1.0X
+ParquetReader Vectorized: DataPageV2 83 88 10 189.1 5.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 57 59 3 275.7 3.6 1.8X
+ParquetReader Vectorized -> Row: DataPageV2 38 40 3 416.3 2.4 2.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12263 12285 31 1.3 779.6 1.0X
+SQL Json 9495 9501 9 1.7 603.6 1.3X
+SQL Parquet Vectorized: DataPageV1 162 175 10 97.1 10.3 75.7X
+SQL Parquet Vectorized: DataPageV2 161 172 12 97.9 10.2 76.4X
+SQL Parquet MR: DataPageV1 2074 2105 44 7.6 131.9 5.9X
+SQL Parquet MR: DataPageV2 1974 1981 9 8.0 125.5 6.2X
+SQL ORC Vectorized 187 218 30 84.3 11.9 65.7X
+SQL ORC MR 1529 1553 34 10.3 97.2 8.0X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 205 214 12 76.8 13.0 1.0X
+ParquetReader Vectorized: DataPageV2 204 211 10 77.2 13.0 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 122 132 24 128.7 7.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 122 126 6 128.4 7.8 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12946 12955 12 1.2 823.1 1.0X
+SQL Json 9871 9876 8 1.6 627.6 1.3X
+SQL Parquet Vectorized: DataPageV1 157 200 34 100.0 10.0 82.3X
+SQL Parquet Vectorized: DataPageV2 229 242 14 68.8 14.5 56.7X
+SQL Parquet MR: DataPageV1 2388 2389 2 6.6 151.8 5.4X
+SQL Parquet MR: DataPageV2 2080 2087 10 7.6 132.2 6.2X
+SQL ORC Vectorized 240 285 23 65.6 15.2 54.0X
+SQL ORC MR 1699 1732 46 9.3 108.0 7.6X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 242 245 5 65.0 15.4 1.0X
+ParquetReader Vectorized: DataPageV2 309 314 9 50.9 19.7 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 227 268 18 69.3 14.4 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 294 312 25 53.5 18.7 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14331 14347 22 1.1 911.2 1.0X
+SQL Json 10406 10434 40 1.5 661.6 1.4X
+SQL Parquet Vectorized: DataPageV1 153 196 41 102.7 9.7 93.6X
+SQL Parquet Vectorized: DataPageV2 378 415 30 41.6 24.0 37.9X
+SQL Parquet MR: DataPageV1 2439 2446 11 6.4 155.1 5.9X
+SQL Parquet MR: DataPageV2 2181 2188 10 7.2 138.7 6.6X
+SQL ORC Vectorized 320 346 25 49.2 20.3 44.8X
+SQL ORC MR 1851 1853 3 8.5 117.7 7.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 258 262 9 60.9 16.4 1.0X
+ParquetReader Vectorized: DataPageV2 481 484 3 32.7 30.6 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 250 275 26 62.9 15.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 475 502 27 33.1 30.2 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 18450 18451 2 0.9 1173.0 1.0X
+SQL Json 12553 12562 13 1.3 798.1 1.5X
+SQL Parquet Vectorized: DataPageV1 259 272 12 60.8 16.5 71.3X
+SQL Parquet Vectorized: DataPageV2 534 566 22 29.4 34.0 34.5X
+SQL Parquet MR: DataPageV1 2529 2537 11 6.2 160.8 7.3X
+SQL Parquet MR: DataPageV2 2331 2334 4 6.7 148.2 7.9X
+SQL ORC Vectorized 424 460 36 37.1 27.0 43.5X
+SQL ORC MR 2009 2023 20 7.8 127.7 9.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 322 338 21 48.9 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 674 683 12 23.3 42.9 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 352 358 9 44.7 22.4 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 628 660 22 25.0 39.9 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14806 14816 14 1.1 941.3 1.0X
+SQL Json 11968 11969 1 1.3 760.9 1.2X
+SQL Parquet Vectorized: DataPageV1 150 184 26 105.0 9.5 98.8X
+SQL Parquet Vectorized: DataPageV2 147 183 32 107.2 9.3 100.9X
+SQL Parquet MR: DataPageV1 2338 2352 19 6.7 148.7 6.3X
+SQL Parquet MR: DataPageV2 2221 2267 65 7.1 141.2 6.7X
+SQL ORC Vectorized 475 494 29 33.1 30.2 31.1X
+SQL ORC MR 1967 1978 16 8.0 125.1 7.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 236 241 8 66.7 15.0 1.0X
+ParquetReader Vectorized: DataPageV2 237 241 9 66.3 15.1 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 218 244 25 72.1 13.9 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 218 251 21 72.2 13.8 1.1X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 19335 19346 15 0.8 1229.3 1.0X
+SQL Json 16112 16121 13 1.0 1024.4 1.2X
+SQL Parquet Vectorized: DataPageV1 257 278 29 61.1 16.4 75.1X
+SQL Parquet Vectorized: DataPageV2 258 268 9 60.9 16.4 74.9X
+SQL Parquet MR: DataPageV1 2542 2557 20 6.2 161.6 7.6X
+SQL Parquet MR: DataPageV2 2416 2439 32 6.5 153.6 8.0X
+SQL ORC Vectorized 593 605 18 26.5 37.7 32.6X
+SQL ORC MR 2134 2141 11 7.4 135.7 9.1X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 322 346 24 48.8 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 326 326 1 48.3 20.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 350 359 9 44.9 22.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 348 358 10 45.2 22.1 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 13899 14000 142 0.8 1325.5 1.0X
+SQL Json 11275 11289 20 0.9 1075.3 1.2X
+SQL Parquet Vectorized: DataPageV1 2092 2107 21 5.0 199.5 6.6X
+SQL Parquet Vectorized: DataPageV2 3073 3074 2 3.4 293.0 4.5X
+SQL Parquet MR: DataPageV1 4192 4212 29 2.5 399.8 3.3X
+SQL Parquet MR: DataPageV2 4133 4194 87 2.5 394.1 3.4X
+SQL ORC Vectorized 2218 2219 1 4.7 211.5 6.3X
+SQL ORC MR 3767 3776 12 2.8 359.3 3.7X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 7367 7387 28 1.4 702.6 1.0X
+SQL Json 6817 6817 0 1.5 650.1 1.1X
+SQL Parquet Vectorized: DataPageV1 602 618 15 17.4 57.5 12.2X
+SQL Parquet Vectorized: DataPageV2 599 610 15 17.5 57.1 12.3X
+SQL Parquet MR: DataPageV1 1888 1936 68 5.6 180.0 3.9X
+SQL Parquet MR: DataPageV2 2000 2018 25 5.2 190.7 3.7X
+SQL ORC Vectorized 527 545 22 19.9 50.2 14.0X
+SQL ORC MR 1916 1927 16 5.5 182.7 3.8X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 19239 19257 25 0.8 1223.2 1.0X
+Data column - Json 12387 12393 8 1.3 787.6 1.6X
+Data column - Parquet Vectorized: DataPageV1 227 269 25 69.2 14.5 84.6X
+Data column - Parquet Vectorized: DataPageV2 612 651 28 25.7 38.9 31.4X
+Data column - Parquet MR: DataPageV1 2989 3016 39 5.3 190.0 6.4X
+Data column - Parquet MR: DataPageV2 2750 2754 5 5.7 174.8 7.0X
+Data column - ORC Vectorized 426 467 33 37.0 27.1 45.2X
+Data column - ORC MR 2513 2538 35 6.3 159.8 7.7X
+Partition column - CSV 6623 6627 5 2.4 421.1 2.9X
+Partition column - Json 10234 10235 2 1.5 650.7 1.9X
+Partition column - Parquet Vectorized: DataPageV1 56 73 19 279.8 3.6 342.2X
+Partition column - Parquet Vectorized: DataPageV2 57 72 19 278.3 3.6 340.4X
+Partition column - Parquet MR: DataPageV1 1392 1417 36 11.3 88.5 13.8X
+Partition column - Parquet MR: DataPageV2 1390 1416 37 11.3 88.4 13.8X
+Partition column - ORC Vectorized 56 89 36 283.2 3.5 346.4X
+Partition column - ORC MR 1578 1581 4 10.0 100.4 12.2X
+Both columns - CSV 19178 19181 4 0.8 1219.3 1.0X
+Both columns - Json 13104 13105 1 1.2 833.1 1.5X
+Both columns - Parquet Vectorized: DataPageV1 314 338 21 50.2 19.9 61.4X
+Both columns - Parquet Vectorized: DataPageV2 708 741 54 22.2 45.0 27.2X
+Both columns - Parquet MR: DataPageV1 3083 3105 31 5.1 196.0 6.2X
+Both columns - Parquet MR: DataPageV2 2897 2901 6 5.4 184.2 6.6X
+Both columns - ORC Vectorized 456 504 39 34.5 29.0 42.1X
+Both columns - ORC MR 2594 2597 4 6.1 164.9 7.4X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 9165 9257 130 1.1 874.1 1.0X
+SQL Json 10230 10234 7 1.0 975.6 0.9X
+SQL Parquet Vectorized: DataPageV1 1275 1315 56 8.2 121.6 7.2X
+SQL Parquet Vectorized: DataPageV2 2406 2407 0 4.4 229.5 3.8X
+SQL Parquet MR: DataPageV1 4005 4009 7 2.6 381.9 2.3X
+SQL Parquet MR: DataPageV2 4358 4366 12 2.4 415.6 2.1X
+ParquetReader Vectorized: DataPageV1 985 995 13 10.6 94.0 9.3X
+ParquetReader Vectorized: DataPageV2 2039 2061 32 5.1 194.4 4.5X
+SQL ORC Vectorized 1048 1072 34 10.0 99.9 8.7X
+SQL ORC MR 3179 3196 24 3.3 303.2 2.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 6749 6750 2 1.6 643.6 1.0X
+SQL Json 7960 7967 10 1.3 759.1 0.8X
+SQL Parquet Vectorized: DataPageV1 1078 1105 37 9.7 102.8 6.3X
+SQL Parquet Vectorized: DataPageV2 1939 1941 3 5.4 184.9 3.5X
+SQL Parquet MR: DataPageV1 3090 3099 13 3.4 294.7 2.2X
+SQL Parquet MR: DataPageV2 3274 3286 17 3.2 312.3 2.1X
+ParquetReader Vectorized: DataPageV1 1058 1067 13 9.9 100.9 6.4X
+ParquetReader Vectorized: DataPageV2 1847 1848 2 5.7 176.2 3.7X
+SQL ORC Vectorized 1307 1307 0 8.0 124.6 5.2X
+SQL ORC MR 3078 3122 62 3.4 293.6 2.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 4818 4824 8 2.2 459.5 1.0X
+SQL Json 4853 4878 35 2.2 462.8 1.0X
+SQL Parquet Vectorized: DataPageV1 255 264 6 41.1 24.3 18.9X
+SQL Parquet Vectorized: DataPageV2 711 716 4 14.7 67.8 6.8X
Review comment:
Seems it is related to the percentage of nulls. BTW do you see any benefit from switching to new implementation for `DELTA_BYTE_ARRAY`? The old implementation just relies on `parquet-mr` and is very simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r790027843
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -283,25 +294,30 @@ private void initDataReader(
} catch (IOException e) {
throw new IOException("could not read page in col " + descriptor, e);
}
+ if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding) &&
+ previousReader != null && previousReader instanceof RequiresPreviousReader) {
Review comment:
You're right, it is not needed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java
##########
@@ -86,4 +87,26 @@ void readLongsWithRebase(
void write(WritableColumnVector outputColumnVector, int rowId, long val);
}
+ @FunctionalInterface
+ interface ByteBufferOutputWriter {
+ void write(WritableColumnVector c, int rowId, ByteBuffer val, int length);
+
+ static void writeArrayByteBuffer(WritableColumnVector c, int rowId, ByteBuffer val,
Review comment:
I don't know if it is frowned upon. In this case, not including in the interface only leads to some code bloat.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
Review comment:
done
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
Review comment:
Nope it doesn't. Removed
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,133 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ Binary binaryVal;
+
+ VectorizedDeltaByteArrayReader(MemoryMode memoryMode){
+ this.memoryMode = memoryMode;
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader(memoryMode);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ this.valueCount = valueCount;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ prefixLengthVector = new OffHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OffHeapColumnVector(valueCount, BinaryType);
+ } else {
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ }
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, null, 0,
+ (w, r, v, l) ->
Review comment:
I really hope not. AFAIK, lambdas are highly optimized to not incur object creation overhead. I'm not sure if the function call overhead might also be eliminated by inlining.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r792233164
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private int valueCount;
Review comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1021741123
Updated the JDK 8 benchmark results as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r793451912
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
Review comment:
Maybe we should call `lengthsVector.putInts(0, valueCount, 0);` to ensure this init value of OffHeapColumnVector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra edited a comment on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1022462107
> > Updated the JDK 8 benchmark results as well.
>
> After comparing the new bench data, I find that the data corresponding to `Parquet Data Page V2` in the two test cases `String with Nulls Scan (50.0%) ` and `String with Nulls Scan (95.0%)` is relatively slower than the previous pr (although the CPU frequency of the testing machine is reduced):
>
> before after
> String with Nulls Scan (50.0%) 145.7 ns/per row 228.7 ns/per row
> String with Nulls Scan (95.0%) 25.2 ns/per row 77.9 ns/per row
It's hard to reasonably compare the numbers across runs (even though the difference is substantial) because of the difference in the environment.
Incidentally, with nulls, the decoder doesn't even get called so such a precipitous drop is somewhat suspicious. _And_ it appears that the vectorized decoder is being called one record at a time (this may not be a problem because the decoding has mostly been done though not written into the output vector).
I made a change to determine runs of null/non-null values and increase the number of values being written out to the output vector in each call, but saw no significant change (running benchmark on laptop).
See: https://github.com/apache/spark/blob/6e64e9252a821651a8984babfaccccc79a9ea433/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java#L237
Let me do a profile run to see if any obvious bottlenecks stand out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r796077542
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
Added the check. Locally run benchmark showed no real gain though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r789359548
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -283,25 +294,30 @@ private void initDataReader(
} catch (IOException e) {
throw new IOException("could not read page in col " + descriptor, e);
}
+ if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding) &&
+ previousReader != null && previousReader instanceof RequiresPreviousReader) {
Review comment:
Is `previousReader != null` necessary?
`previousReader instanceof RequiresPreviousReader` can covered `previousReader != null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on pull request #35262:
URL: https://github.com/apache/spark/pull/35262#issuecomment-1025294093
> When `memoryMode` is `MemoryMode.OFF_HEAP`, some `OffHeapColumnVector` will be create like `binaryValVector`, `lengthsVector` and so on. `OffHeapColumnVector` needs to call the `close` method manually to release the allocated memory, but I can't find the corresponding call.
Oh dear. This will take some fixing. Thank you for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r794236177
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final MemoryMode memoryMode;
+ private final VectorizedDeltaBinaryPackedReader lengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader(MemoryMode memoryMode) {
+ this.memoryMode = memoryMode;
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ lengthsVector = new OffHeapColumnVector(valueCount, IntegerType);
+ lengthsVector.putInts(0, valueCount, 0);
+ } else {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ }
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ if (total == 0) {
+ return;
+ }
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter;
+ if (memoryMode == MemoryMode.OFF_HEAP) {
+ outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
+ } else {
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ }
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
Review comment:
@parthchandra if `This method gets called only for non-null values.`, then maybe there should be no need to add `lengthsVector.putInts(0, valueCount, 0)` because the place where the exception was thrown was originally a null value when running `String with Nulls Scan`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r797366940
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -66,20 +57,18 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
}
ByteBuffer buffer;
ByteBufferOutputWriter outputWriter;
- if (memoryMode == MemoryMode.OFF_HEAP) {
- outputWriter = ByteBufferOutputWriter::copyWriteByteBuffer;
- } else {
- outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
- }
+ outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
int length;
for (int i = 0; i < total; i++) {
length = lengthsVector.getInt(rowId + i);
- try {
- buffer = in.slice(length);
- } catch (EOFException e) {
- throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ if (length > 0) {
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
}
- outputWriter.write(c, rowId + i, buffer, length);
currentRow++;
Review comment:
should we change line72 to `currentRow += total` and move it out of the loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r797411267
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
+ suffixVector = new OnHeapColumnVector(valueCount, BinaryType);
+ prefixLengthReader.initFromPage(valueCount, in);
+ prefixLengthReader.readIntegers(prefixLengthReader.getTotalValueCount(),
+ prefixLengthVector, 0);
+ suffixReader.initFromPage(valueCount, in);
+ suffixReader.readBinary(valueCount, suffixVector, 0);
}
@Override
public Binary readBinary(int len) {
- return deltaByteArrayReader.readBytes();
+ readValues(1, binaryValVector, 0, ByteBufferOutputWriter::writeArrayByteBuffer);
+ return Binary.fromConstantByteArray(binaryValVector.getBinary(0));
}
- @Override
- public void readBinary(int total, WritableColumnVector c, int rowId) {
+ public void readValues(int total, WritableColumnVector c, int rowId,
Review comment:
public or private? I found that no other classes use this API directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r810298357
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 14859 14914 77 1.1 944.7 1.0X
+SQL Json 9418 9457 55 1.7 598.8 1.6X
+SQL Parquet Vectorized: DataPageV1 109 128 14 144.6 6.9 136.6X
+SQL Parquet Vectorized: DataPageV2 79 89 8 199.3 5.0 188.3X
+SQL Parquet MR: DataPageV1 1699 1743 62 9.3 108.0 8.7X
+SQL Parquet MR: DataPageV2 1462 1489 38 10.8 93.0 10.2X
+SQL ORC Vectorized 165 200 33 95.3 10.5 90.0X
+SQL ORC MR 1409 1420 16 11.2 89.6 10.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 101 104 3 155.2 6.4 1.0X
+ParquetReader Vectorized: DataPageV2 82 85 5 192.0 5.2 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 48 50 2 324.6 3.1 2.1X
+ParquetReader Vectorized -> Row: DataPageV2 29 31 3 539.4 1.9 3.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 17619 17639 28 0.9 1120.2 1.0X
+SQL Json 10590 10606 23 1.5 673.3 1.7X
+SQL Parquet Vectorized: DataPageV1 178 194 10 88.2 11.3 98.8X
+SQL Parquet Vectorized: DataPageV2 178 188 9 88.2 11.3 98.7X
+SQL Parquet MR: DataPageV1 1884 1887 4 8.4 119.8 9.4X
+SQL Parquet MR: DataPageV2 1689 1742 75 9.3 107.4 10.4X
+SQL ORC Vectorized 162 193 24 97.0 10.3 108.7X
+SQL ORC MR 1505 1552 67 10.5 95.7 11.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 230 236 13 68.3 14.6 1.0X
+ParquetReader Vectorized: DataPageV2 228 233 8 69.1 14.5 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 138 150 26 113.7 8.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 137 140 2 114.5 8.7 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 18488 18494 8 0.9 1175.5 1.0X
+SQL Json 11190 11195 7 1.4 711.4 1.7X
+SQL Parquet Vectorized: DataPageV1 125 155 34 125.7 8.0 147.7X
+SQL Parquet Vectorized: DataPageV2 183 192 9 86.1 11.6 101.2X
+SQL Parquet MR: DataPageV1 2153 2160 10 7.3 136.9 8.6X
+SQL Parquet MR: DataPageV2 1876 1889 18 8.4 119.3 9.9X
+SQL ORC Vectorized 212 257 23 74.4 13.4 87.4X
+SQL ORC MR 1653 1658 7 9.5 105.1 11.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 198 201 5 79.6 12.6 1.0X
+ParquetReader Vectorized: DataPageV2 256 260 3 61.5 16.3 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 193 226 14 81.4 12.3 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 250 253 2 62.8 15.9 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 18614 18703 125 0.8 1183.5 1.0X
+SQL Json 11673 11711 53 1.3 742.2 1.6X
+SQL Parquet Vectorized: DataPageV1 128 154 26 123.1 8.1 145.7X
+SQL Parquet Vectorized: DataPageV2 270 302 23 58.3 17.1 69.0X
+SQL Parquet MR: DataPageV1 2117 2145 39 7.4 134.6 8.8X
+SQL Parquet MR: DataPageV2 1855 1860 7 8.5 117.9 10.0X
+SQL ORC Vectorized 277 292 16 56.7 17.6 67.2X
+SQL ORC MR 1623 1629 9 9.7 103.2 11.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 225 226 1 69.9 14.3 1.0X
+ParquetReader Vectorized: DataPageV2 362 365 2 43.4 23.0 0.6X
+ParquetReader Vectorized -> Row: DataPageV1 193 218 18 81.5 12.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV2 360 366 6 43.7 22.9 0.6X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 23939 23953 20 0.7 1522.0 1.0X
+SQL Json 14445 14449 5 1.1 918.4 1.7X
+SQL Parquet Vectorized: DataPageV1 186 229 28 84.7 11.8 128.9X
+SQL Parquet Vectorized: DataPageV2 459 493 25 34.3 29.2 52.2X
+SQL Parquet MR: DataPageV1 2180 2184 7 7.2 138.6 11.0X
+SQL Parquet MR: DataPageV2 1954 1973 27 8.1 124.2 12.3X
+SQL ORC Vectorized 368 392 24 42.8 23.4 65.1X
+SQL ORC MR 1793 1794 2 8.8 114.0 13.4X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 280 293 18 56.1 17.8 1.0X
+ParquetReader Vectorized: DataPageV2 577 602 48 27.3 36.7 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 314 321 10 50.1 19.9 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 581 584 4 27.1 37.0 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 19278 19291 18 0.8 1225.6 1.0X
+SQL Json 13366 13381 21 1.2 849.8 1.4X
+SQL Parquet Vectorized: DataPageV1 130 152 23 120.8 8.3 148.1X
+SQL Parquet Vectorized: DataPageV2 135 157 17 116.8 8.6 143.2X
+SQL Parquet MR: DataPageV1 2126 2137 15 7.4 135.2 9.1X
+SQL Parquet MR: DataPageV2 1970 1985 21 8.0 125.2 9.8X
+SQL ORC Vectorized 387 396 11 40.7 24.6 49.8X
+SQL ORC MR 1831 1832 1 8.6 116.4 10.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 194 197 5 81.1 12.3 1.0X
+ParquetReader Vectorized: DataPageV2 194 197 7 81.2 12.3 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 225 253 18 69.9 14.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 224 252 18 70.2 14.2 0.9X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 24691 24705 19 0.6 1569.8 1.0X
+SQL Json 18028 18028 0 0.9 1146.2 1.4X
+SQL Parquet Vectorized: DataPageV1 190 225 28 83.0 12.0 130.3X
+SQL Parquet Vectorized: DataPageV2 188 230 26 83.9 11.9 131.7X
+SQL Parquet MR: DataPageV1 2362 2365 4 6.7 150.2 10.5X
+SQL Parquet MR: DataPageV2 2061 2078 25 7.6 131.0 12.0X
+SQL ORC Vectorized 499 524 37 31.6 31.7 49.5X
+SQL ORC MR 1870 1880 14 8.4 118.9 13.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 276 295 21 57.0 17.5 1.0X
+ParquetReader Vectorized: DataPageV2 278 289 17 56.6 17.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 315 326 15 50.0 20.0 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 315 323 8 49.9 20.0 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 16840 16908 96 0.6 1606.0 1.0X
+SQL Json 12496 12513 25 0.8 1191.7 1.3X
+SQL Parquet Vectorized: DataPageV1 2169 2172 5 4.8 206.9 7.8X
+SQL Parquet Vectorized: DataPageV2 3102 3119 24 3.4 295.9 5.4X
+SQL Parquet MR: DataPageV1 4140 4144 5 2.5 394.8 4.1X
+SQL Parquet MR: DataPageV2 3988 3996 12 2.6 380.3 4.2X
+SQL ORC Vectorized 2180 2196 23 4.8 207.9 7.7X
+SQL ORC MR 3765 3766 2 2.8 359.0 4.5X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 9960 9960 0 1.1 949.8 1.0X
+SQL Json 7625 7712 123 1.4 727.2 1.3X
+SQL Parquet Vectorized: DataPageV1 577 582 6 18.2 55.0 17.3X
+SQL Parquet Vectorized: DataPageV2 584 592 6 18.0 55.7 17.1X
+SQL Parquet MR: DataPageV1 1722 1736 19 6.1 164.2 5.8X
+SQL Parquet MR: DataPageV2 1662 1668 9 6.3 158.5 6.0X
+SQL ORC Vectorized 483 524 27 21.7 46.1 20.6X
+SQL ORC MR 1841 1850 14 5.7 175.5 5.4X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 23787 23788 2 0.7 1512.3 1.0X
+Data column - Json 13993 14011 25 1.1 889.7 1.7X
+Data column - Parquet Vectorized: DataPageV1 184 235 36 85.4 11.7 129.2X
+Data column - Parquet Vectorized: DataPageV2 531 542 15 29.6 33.7 44.8X
+Data column - Parquet MR: DataPageV1 2539 2547 13 6.2 161.4 9.4X
+Data column - Parquet MR: DataPageV2 2299 2301 3 6.8 146.2 10.3X
+Data column - ORC Vectorized 379 403 23 41.5 24.1 62.8X
+Data column - ORC MR 2047 2070 33 7.7 130.1 11.6X
+Partition column - CSV 6834 6835 1 2.3 434.5 3.5X
+Partition column - Json 11444 11478 49 1.4 727.6 2.1X
+Partition column - Parquet Vectorized: DataPageV1 51 71 22 308.6 3.2 466.7X
+Partition column - Parquet Vectorized: DataPageV2 51 61 16 310.5 3.2 469.5X
+Partition column - Parquet MR: DataPageV1 1203 1214 15 13.1 76.5 19.8X
+Partition column - Parquet MR: DataPageV2 1210 1224 20 13.0 76.9 19.7X
+Partition column - ORC Vectorized 52 67 14 303.1 3.3 458.4X
+Partition column - ORC MR 1338 1342 5 11.8 85.1 17.8X
+Both columns - CSV 24051 24052 2 0.7 1529.1 1.0X
+Both columns - Json 15016 15030 20 1.0 954.7 1.6X
+Both columns - Parquet Vectorized: DataPageV1 235 269 27 66.9 15.0 101.2X
+Both columns - Parquet Vectorized: DataPageV2 563 617 60 27.9 35.8 42.2X
+Both columns - Parquet MR: DataPageV1 2525 2555 43 6.2 160.5 9.4X
+Both columns - Parquet MR: DataPageV2 2256 2267 15 7.0 143.5 10.5X
+Both columns - ORC Vectorized 407 454 51 38.7 25.9 58.5X
+Both columns - ORC MR 2153 2155 2 7.3 136.9 11.0X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 11812 11849 53 0.9 1126.4 1.0X
+SQL Json 11454 11467 18 0.9 1092.3 1.0X
+SQL Parquet Vectorized: DataPageV1 1250 1276 37 8.4 119.2 9.5X
+SQL Parquet Vectorized: DataPageV2 2248 2261 17 4.7 214.4 5.3X
+SQL Parquet MR: DataPageV1 3629 3630 1 2.9 346.1 3.3X
+SQL Parquet MR: DataPageV2 3929 3934 6 2.7 374.7 3.0X
+ParquetReader Vectorized: DataPageV1 921 922 2 11.4 87.8 12.8X
+ParquetReader Vectorized: DataPageV2 1890 1890 0 5.5 180.3 6.2X
+SQL ORC Vectorized 1079 1105 36 9.7 102.9 10.9X
+SQL ORC MR 3042 3070 40 3.4 290.1 3.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+SQL CSV 8399 8410 16 1.2 801.0 1.0X
+SQL Json 8892 8905 18 1.2 848.0 0.9X
+SQL Parquet Vectorized: DataPageV1 1065 1092 38 9.8 101.6 7.9X
+SQL Parquet Vectorized: DataPageV2 1747 1747 0 6.0 166.6 4.8X
+SQL Parquet MR: DataPageV1 2718 2719 1 3.9 259.2 3.1X
+SQL Parquet MR: DataPageV2 2955 2964 12 3.5 281.8 2.8X
+ParquetReader Vectorized: DataPageV1 1082 1084 3 9.7 103.2 7.8X
+ParquetReader Vectorized: DataPageV2 1707 1713 9 6.1 162.8 4.9X
+SQL ORC Vectorized 1345 1357 17 7.8 128.3 6.2X
+SQL ORC MR 3012 3046 47 3.5 287.3 2.8X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1028-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 6169 6176 10 1.7 588.3 1.0X
+SQL Json 5352 5376 35 2.0 510.4 1.2X
+SQL Parquet Vectorized: DataPageV1 248 255 7 42.3 23.6 24.9X
+SQL Parquet Vectorized: DataPageV2 364 372 10 28.8 34.7 17.0X
+SQL Parquet MR: DataPageV1 1624 1626 3 6.5 154.9 3.8X
+SQL Parquet MR: DataPageV2 1520 1526 8 6.9 145.0 4.1X
+ParquetReader Vectorized: DataPageV1 259 262 1 40.4 24.7 23.8X
+ParquetReader Vectorized: DataPageV2 376 378 2 27.9 35.9 16.4X
Review comment:
From the code I linked above, the vectorized reader doesn't even get called for null values. So in the 95% nulls case, the new code is mostly not even called. Wonder if we are hitting a branch prediction miss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r802222581
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java
##########
@@ -16,51 +16,121 @@
*/
package org.apache.spark.sql.execution.datasources.parquet;
+import static org.apache.spark.sql.types.DataTypes.BinaryType;
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader;
+import org.apache.parquet.column.values.RequiresPreviousReader;
+import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.api.Binary;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
- * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized interface.
+ * An implementation of the Parquet DELTA_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
*/
-public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase {
- private final DeltaByteArrayReader deltaByteArrayReader = new DeltaByteArrayReader();
+public class VectorizedDeltaByteArrayReader extends VectorizedReaderBase
+ implements VectorizedValuesReader, RequiresPreviousReader {
+
+ private final VectorizedDeltaBinaryPackedReader prefixLengthReader =
+ new VectorizedDeltaBinaryPackedReader();
+ private final VectorizedDeltaLengthByteArrayReader suffixReader;
+ private WritableColumnVector prefixLengthVector;
+ private WritableColumnVector suffixVector;
+ private byte[] previous = new byte[0];
+ private int currentRow = 0;
+
+ //temporary variable used by getBinary
+ private final WritableColumnVector binaryValVector;
+
+ VectorizedDeltaByteArrayReader() {
+ this.suffixReader = new VectorizedDeltaLengthByteArrayReader();
+ binaryValVector = new OnHeapColumnVector(1, BinaryType);
+ }
@Override
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
- deltaByteArrayReader.initFromPage(valueCount, in);
+ prefixLengthVector = new OnHeapColumnVector(valueCount, IntegerType);
Review comment:
As discussed earlier (https://github.com/apache/spark/pull/35262#issuecomment-1025369374), if we use `OffHeapColumnVector` here, we need to consider the memory leak issue because there is no path to call `OffHeapColumnVector.close()` method currently, this will make the current pr more complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r804277234
##########
File path: sql/core/benchmarks/DataSourceReadBenchmark-jdk11-results.txt
##########
@@ -2,322 +2,322 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 9636 9771 191 1.6 612.6 1.0X
-SQL Json 7960 8227 378 2.0 506.1 1.2X
-SQL Parquet Vectorized: DataPageV1 113 129 12 139.7 7.2 85.6X
-SQL Parquet Vectorized: DataPageV2 84 93 12 186.6 5.4 114.3X
-SQL Parquet MR: DataPageV1 1466 1470 6 10.7 93.2 6.6X
-SQL Parquet MR: DataPageV2 1334 1347 18 11.8 84.8 7.2X
-SQL ORC Vectorized 163 197 27 96.3 10.4 59.0X
-SQL ORC MR 1554 1558 6 10.1 98.8 6.2X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 10153 10161 12 1.5 645.5 1.0X
+SQL Json 8463 8512 69 1.9 538.0 1.2X
+SQL Parquet Vectorized: DataPageV1 131 149 14 120.0 8.3 77.5X
+SQL Parquet Vectorized: DataPageV2 98 112 15 161.2 6.2 104.0X
+SQL Parquet MR: DataPageV1 1968 1968 0 8.0 125.1 5.2X
+SQL Parquet MR: DataPageV2 1735 1739 6 9.1 110.3 5.9X
+SQL ORC Vectorized 164 198 41 96.0 10.4 62.0X
+SQL ORC MR 1572 1581 12 10.0 100.0 6.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BOOLEAN Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 94 103 13 167.1 6.0 1.0X
-ParquetReader Vectorized: DataPageV2 77 86 11 204.3 4.9 1.2X
-ParquetReader Vectorized -> Row: DataPageV1 44 47 4 357.0 2.8 2.1X
-ParquetReader Vectorized -> Row: DataPageV2 35 37 3 445.2 2.2 2.7X
+ParquetReader Vectorized: DataPageV1 102 107 14 154.6 6.5 1.0X
+ParquetReader Vectorized: DataPageV2 83 88 10 189.1 5.3 1.2X
+ParquetReader Vectorized -> Row: DataPageV1 57 59 3 275.7 3.6 1.8X
+ParquetReader Vectorized -> Row: DataPageV2 38 40 3 416.3 2.4 2.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 11479 11919 622 1.4 729.8 1.0X
-SQL Json 9894 9922 39 1.6 629.1 1.2X
-SQL Parquet Vectorized: DataPageV1 123 156 30 128.3 7.8 93.6X
-SQL Parquet Vectorized: DataPageV2 126 138 19 125.2 8.0 91.4X
-SQL Parquet MR: DataPageV1 1986 2500 726 7.9 126.3 5.8X
-SQL Parquet MR: DataPageV2 1810 1898 126 8.7 115.1 6.3X
-SQL ORC Vectorized 174 210 30 90.5 11.0 66.1X
-SQL ORC MR 1645 1652 9 9.6 104.6 7.0X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12263 12285 31 1.3 779.6 1.0X
+SQL Json 9495 9501 9 1.7 603.6 1.3X
+SQL Parquet Vectorized: DataPageV1 162 175 10 97.1 10.3 75.7X
+SQL Parquet Vectorized: DataPageV2 161 172 12 97.9 10.2 76.4X
+SQL Parquet MR: DataPageV1 2074 2105 44 7.6 131.9 5.9X
+SQL Parquet MR: DataPageV2 1974 1981 9 8.0 125.5 6.2X
+SQL ORC Vectorized 187 218 30 84.3 11.9 65.7X
+SQL ORC MR 1529 1553 34 10.3 97.2 8.0X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 166 177 14 94.9 10.5 1.0X
-ParquetReader Vectorized: DataPageV2 165 172 11 95.3 10.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 95 100 5 165.7 6.0 1.7X
-ParquetReader Vectorized -> Row: DataPageV2 85 89 6 186.0 5.4 2.0X
+ParquetReader Vectorized: DataPageV1 205 214 12 76.8 13.0 1.0X
+ParquetReader Vectorized: DataPageV2 204 211 10 77.2 13.0 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 122 132 24 128.7 7.8 1.7X
+ParquetReader Vectorized -> Row: DataPageV2 122 126 6 128.4 7.8 1.7X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12176 12646 664 1.3 774.1 1.0X
-SQL Json 9696 9729 46 1.6 616.5 1.3X
-SQL Parquet Vectorized: DataPageV1 151 201 33 103.9 9.6 80.4X
-SQL Parquet Vectorized: DataPageV2 216 235 15 72.7 13.8 56.3X
-SQL Parquet MR: DataPageV1 1915 2017 145 8.2 121.8 6.4X
-SQL Parquet MR: DataPageV2 1954 1978 33 8.0 124.3 6.2X
-SQL ORC Vectorized 197 235 25 79.7 12.6 61.7X
-SQL ORC MR 1769 1829 85 8.9 112.5 6.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 12946 12955 12 1.2 823.1 1.0X
+SQL Json 9871 9876 8 1.6 627.6 1.3X
+SQL Parquet Vectorized: DataPageV1 157 200 34 100.0 10.0 82.3X
+SQL Parquet Vectorized: DataPageV2 229 242 14 68.8 14.5 56.7X
+SQL Parquet MR: DataPageV1 2388 2389 2 6.6 151.8 5.4X
+SQL Parquet MR: DataPageV2 2080 2087 10 7.6 132.2 6.2X
+SQL ORC Vectorized 240 285 23 65.6 15.2 54.0X
+SQL ORC MR 1699 1732 46 9.3 108.0 7.6X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 230 237 12 68.5 14.6 1.0X
-ParquetReader Vectorized: DataPageV2 293 298 9 53.6 18.7 0.8X
-ParquetReader Vectorized -> Row: DataPageV1 215 265 23 73.2 13.7 1.1X
-ParquetReader Vectorized -> Row: DataPageV2 279 301 32 56.3 17.8 0.8X
+ParquetReader Vectorized: DataPageV1 242 245 5 65.0 15.4 1.0X
+ParquetReader Vectorized: DataPageV2 309 314 9 50.9 19.7 0.8X
+ParquetReader Vectorized -> Row: DataPageV1 227 268 18 69.3 14.4 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 294 312 25 53.5 18.7 0.8X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13069 13409 482 1.2 830.9 1.0X
-SQL Json 10599 10621 32 1.5 673.9 1.2X
-SQL Parquet Vectorized: DataPageV1 142 177 34 110.6 9.0 91.9X
-SQL Parquet Vectorized: DataPageV2 313 359 28 50.2 19.9 41.7X
-SQL Parquet MR: DataPageV1 1979 2044 92 7.9 125.8 6.6X
-SQL Parquet MR: DataPageV2 1958 2030 101 8.0 124.5 6.7X
-SQL ORC Vectorized 277 303 21 56.7 17.6 47.1X
-SQL ORC MR 1692 1782 128 9.3 107.6 7.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14331 14347 22 1.1 911.2 1.0X
+SQL Json 10406 10434 40 1.5 661.6 1.4X
+SQL Parquet Vectorized: DataPageV1 153 196 41 102.7 9.7 93.6X
+SQL Parquet Vectorized: DataPageV2 378 415 30 41.6 24.0 37.9X
+SQL Parquet MR: DataPageV1 2439 2446 11 6.4 155.1 5.9X
+SQL Parquet MR: DataPageV2 2181 2188 10 7.2 138.7 6.6X
+SQL ORC Vectorized 320 346 25 49.2 20.3 44.8X
+SQL ORC MR 1851 1853 3 8.5 117.7 7.7X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 253 269 18 62.1 16.1 1.0X
-ParquetReader Vectorized: DataPageV2 1197 1199 3 13.1 76.1 0.2X
-ParquetReader Vectorized -> Row: DataPageV1 273 361 110 57.7 17.3 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 379 438 37 41.5 24.1 0.7X
+ParquetReader Vectorized: DataPageV1 258 262 9 60.9 16.4 1.0X
+ParquetReader Vectorized: DataPageV2 481 484 3 32.7 30.6 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 250 275 26 62.9 15.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 475 502 27 33.1 30.2 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17143 17467 458 0.9 1089.9 1.0X
-SQL Json 11507 12198 977 1.4 731.6 1.5X
-SQL Parquet Vectorized: DataPageV1 238 253 19 66.0 15.2 71.9X
-SQL Parquet Vectorized: DataPageV2 502 567 48 31.3 31.9 34.1X
-SQL Parquet MR: DataPageV1 2333 2335 3 6.7 148.4 7.3X
-SQL Parquet MR: DataPageV2 1948 1972 34 8.1 123.8 8.8X
-SQL ORC Vectorized 389 408 20 40.5 24.7 44.1X
-SQL ORC MR 1726 1817 128 9.1 109.7 9.9X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 18450 18451 2 0.9 1173.0 1.0X
+SQL Json 12553 12562 13 1.3 798.1 1.5X
+SQL Parquet Vectorized: DataPageV1 259 272 12 60.8 16.5 71.3X
+SQL Parquet Vectorized: DataPageV2 534 566 22 29.4 34.0 34.5X
+SQL Parquet MR: DataPageV1 2529 2537 11 6.2 160.8 7.3X
+SQL Parquet MR: DataPageV2 2331 2334 4 6.7 148.2 7.9X
+SQL ORC Vectorized 424 460 36 37.1 27.0 43.5X
+SQL ORC MR 2009 2023 20 7.8 127.7 9.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 289 340 43 54.4 18.4 1.0X
-ParquetReader Vectorized: DataPageV2 572 609 27 27.5 36.4 0.5X
-ParquetReader Vectorized -> Row: DataPageV1 329 353 48 47.8 20.9 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 639 654 18 24.6 40.6 0.5X
+ParquetReader Vectorized: DataPageV1 322 338 21 48.9 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 674 683 12 23.3 42.9 0.5X
+ParquetReader Vectorized -> Row: DataPageV1 352 358 9 44.7 22.4 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 628 660 22 25.0 39.9 0.5X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 13721 13812 129 1.1 872.4 1.0X
-SQL Json 12147 17632 2196 1.3 772.3 1.1X
-SQL Parquet Vectorized: DataPageV1 138 164 25 113.9 8.8 99.4X
-SQL Parquet Vectorized: DataPageV2 151 180 26 104.4 9.6 91.1X
-SQL Parquet MR: DataPageV1 2006 2078 101 7.8 127.6 6.8X
-SQL Parquet MR: DataPageV2 2038 2040 2 7.7 129.6 6.7X
-SQL ORC Vectorized 465 475 10 33.8 29.6 29.5X
-SQL ORC MR 1814 1860 64 8.7 115.4 7.6X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 14806 14816 14 1.1 941.3 1.0X
+SQL Json 11968 11969 1 1.3 760.9 1.2X
+SQL Parquet Vectorized: DataPageV1 150 184 26 105.0 9.5 98.8X
+SQL Parquet Vectorized: DataPageV2 147 183 32 107.2 9.3 100.9X
+SQL Parquet MR: DataPageV1 2338 2352 19 6.7 148.7 6.3X
+SQL Parquet MR: DataPageV2 2221 2267 65 7.1 141.2 6.7X
+SQL ORC Vectorized 475 494 29 33.1 30.2 31.1X
+SQL ORC MR 1967 1978 16 8.0 125.1 7.5X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 275 404 187 57.2 17.5 1.0X
-ParquetReader Vectorized: DataPageV2 275 287 12 57.2 17.5 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 227 265 24 69.2 14.4 1.2X
-ParquetReader Vectorized -> Row: DataPageV2 228 259 28 69.1 14.5 1.2X
+ParquetReader Vectorized: DataPageV1 236 241 8 66.7 15.0 1.0X
+ParquetReader Vectorized: DataPageV2 237 241 9 66.3 15.1 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 218 244 25 72.1 13.9 1.1X
+ParquetReader Vectorized -> Row: DataPageV2 218 251 21 72.2 13.8 1.1X
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 17269 17620 496 0.9 1097.9 1.0X
-SQL Json 15636 15952 447 1.0 994.1 1.1X
-SQL Parquet Vectorized: DataPageV1 238 267 18 66.0 15.1 72.5X
-SQL Parquet Vectorized: DataPageV2 222 260 21 70.9 14.1 77.9X
-SQL Parquet MR: DataPageV1 2418 2457 56 6.5 153.7 7.1X
-SQL Parquet MR: DataPageV2 2194 2207 18 7.2 139.5 7.9X
-SQL ORC Vectorized 519 528 14 30.3 33.0 33.3X
-SQL ORC MR 1760 1770 14 8.9 111.9 9.8X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 19335 19346 15 0.8 1229.3 1.0X
+SQL Json 16112 16121 13 1.0 1024.4 1.2X
+SQL Parquet Vectorized: DataPageV1 257 278 29 61.1 16.4 75.1X
+SQL Parquet Vectorized: DataPageV2 258 268 9 60.9 16.4 74.9X
+SQL Parquet MR: DataPageV1 2542 2557 20 6.2 161.6 7.6X
+SQL Parquet MR: DataPageV2 2416 2439 32 6.5 153.6 8.0X
+SQL ORC Vectorized 593 605 18 26.5 37.7 32.6X
+SQL ORC MR 2134 2141 11 7.4 135.7 9.1X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Parquet Reader Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
-ParquetReader Vectorized: DataPageV1 284 305 30 55.3 18.1 1.0X
-ParquetReader Vectorized: DataPageV2 286 286 1 55.1 18.2 1.0X
-ParquetReader Vectorized -> Row: DataPageV1 325 337 16 48.4 20.6 0.9X
-ParquetReader Vectorized -> Row: DataPageV2 346 361 16 45.5 22.0 0.8X
+ParquetReader Vectorized: DataPageV1 322 346 24 48.8 20.5 1.0X
+ParquetReader Vectorized: DataPageV2 326 326 1 48.3 20.7 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 350 359 9 44.9 22.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 348 358 10 45.2 22.1 0.9X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 12428 12714 405 0.8 1185.2 1.0X
-SQL Json 11088 11251 231 0.9 1057.4 1.1X
-SQL Parquet Vectorized: DataPageV1 1990 1997 10 5.3 189.8 6.2X
-SQL Parquet Vectorized: DataPageV2 2551 2618 95 4.1 243.3 4.9X
-SQL Parquet MR: DataPageV1 3903 3913 15 2.7 372.2 3.2X
-SQL Parquet MR: DataPageV2 3734 3920 263 2.8 356.1 3.3X
-SQL ORC Vectorized 2153 2155 3 4.9 205.3 5.8X
-SQL ORC MR 3485 3549 91 3.0 332.4 3.6X
+SQL CSV 13899 14000 142 0.8 1325.5 1.0X
+SQL Json 11275 11289 20 0.9 1075.3 1.2X
+SQL Parquet Vectorized: DataPageV1 2092 2107 21 5.0 199.5 6.6X
+SQL Parquet Vectorized: DataPageV2 3073 3074 2 3.4 293.0 4.5X
+SQL Parquet MR: DataPageV1 4192 4212 29 2.5 399.8 3.3X
+SQL Parquet MR: DataPageV2 4133 4194 87 2.5 394.1 3.4X
+SQL ORC Vectorized 2218 2219 1 4.7 211.5 6.3X
+SQL ORC MR 3767 3776 12 2.8 359.3 3.7X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7116 7167 72 1.5 678.7 1.0X
-SQL Json 6700 6741 58 1.6 639.0 1.1X
-SQL Parquet Vectorized: DataPageV1 526 556 36 19.9 50.1 13.5X
-SQL Parquet Vectorized: DataPageV2 518 533 15 20.2 49.4 13.7X
-SQL Parquet MR: DataPageV1 1504 1656 216 7.0 143.4 4.7X
-SQL Parquet MR: DataPageV2 1676 1676 1 6.3 159.8 4.2X
-SQL ORC Vectorized 497 518 20 21.1 47.4 14.3X
-SQL ORC MR 1657 1787 183 6.3 158.1 4.3X
+SQL CSV 7367 7387 28 1.4 702.6 1.0X
+SQL Json 6817 6817 0 1.5 650.1 1.1X
+SQL Parquet Vectorized: DataPageV1 602 618 15 17.4 57.5 12.2X
+SQL Parquet Vectorized: DataPageV2 599 610 15 17.5 57.1 12.3X
+SQL Parquet MR: DataPageV1 1888 1936 68 5.6 180.0 3.9X
+SQL Parquet MR: DataPageV2 2000 2018 25 5.2 190.7 3.7X
+SQL ORC Vectorized 527 545 22 19.9 50.2 14.0X
+SQL ORC MR 1916 1927 16 5.5 182.7 3.8X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
-Data column - CSV 18247 18411 232 0.9 1160.1 1.0X
-Data column - Json 10860 11264 571 1.4 690.5 1.7X
-Data column - Parquet Vectorized: DataPageV1 223 274 26 70.6 14.2 81.9X
-Data column - Parquet Vectorized: DataPageV2 537 559 23 29.3 34.1 34.0X
-Data column - Parquet MR: DataPageV1 2411 2517 150 6.5 153.3 7.6X
-Data column - Parquet MR: DataPageV2 2299 2356 81 6.8 146.2 7.9X
-Data column - ORC Vectorized 417 433 11 37.7 26.5 43.8X
-Data column - ORC MR 2107 2178 101 7.5 134.0 8.7X
-Partition column - CSV 6090 6186 136 2.6 387.2 3.0X
-Partition column - Json 9479 9603 176 1.7 602.7 1.9X
-Partition column - Parquet Vectorized: DataPageV1 49 69 28 322.0 3.1 373.6X
-Partition column - Parquet Vectorized: DataPageV2 49 63 23 322.1 3.1 373.7X
-Partition column - Parquet MR: DataPageV1 1200 1225 36 13.1 76.3 15.2X
-Partition column - Parquet MR: DataPageV2 1199 1240 57 13.1 76.3 15.2X
-Partition column - ORC Vectorized 53 77 26 295.0 3.4 342.2X
-Partition column - ORC MR 1287 1346 83 12.2 81.8 14.2X
-Both columns - CSV 17671 18140 663 0.9 1123.5 1.0X
-Both columns - Json 11675 12167 696 1.3 742.3 1.6X
-Both columns - Parquet Vectorized: DataPageV1 298 303 9 52.9 18.9 61.3X
-Both columns - Parquet Vectorized: DataPageV2 541 580 36 29.1 34.4 33.7X
-Both columns - Parquet MR: DataPageV1 2448 2491 60 6.4 155.6 7.5X
-Both columns - Parquet MR: DataPageV2 2303 2352 69 6.8 146.4 7.9X
-Both columns - ORC Vectorized 385 406 25 40.9 24.5 47.4X
-Both columns - ORC MR 2118 2202 120 7.4 134.6 8.6X
+Data column - CSV 19239 19257 25 0.8 1223.2 1.0X
+Data column - Json 12387 12393 8 1.3 787.6 1.6X
+Data column - Parquet Vectorized: DataPageV1 227 269 25 69.2 14.5 84.6X
+Data column - Parquet Vectorized: DataPageV2 612 651 28 25.7 38.9 31.4X
+Data column - Parquet MR: DataPageV1 2989 3016 39 5.3 190.0 6.4X
+Data column - Parquet MR: DataPageV2 2750 2754 5 5.7 174.8 7.0X
+Data column - ORC Vectorized 426 467 33 37.0 27.1 45.2X
+Data column - ORC MR 2513 2538 35 6.3 159.8 7.7X
+Partition column - CSV 6623 6627 5 2.4 421.1 2.9X
+Partition column - Json 10234 10235 2 1.5 650.7 1.9X
+Partition column - Parquet Vectorized: DataPageV1 56 73 19 279.8 3.6 342.2X
+Partition column - Parquet Vectorized: DataPageV2 57 72 19 278.3 3.6 340.4X
+Partition column - Parquet MR: DataPageV1 1392 1417 36 11.3 88.5 13.8X
+Partition column - Parquet MR: DataPageV2 1390 1416 37 11.3 88.4 13.8X
+Partition column - ORC Vectorized 56 89 36 283.2 3.5 346.4X
+Partition column - ORC MR 1578 1581 4 10.0 100.4 12.2X
+Both columns - CSV 19178 19181 4 0.8 1219.3 1.0X
+Both columns - Json 13104 13105 1 1.2 833.1 1.5X
+Both columns - Parquet Vectorized: DataPageV1 314 338 21 50.2 19.9 61.4X
+Both columns - Parquet Vectorized: DataPageV2 708 741 54 22.2 45.0 27.2X
+Both columns - Parquet MR: DataPageV1 3083 3105 31 5.1 196.0 6.2X
+Both columns - Parquet MR: DataPageV2 2897 2901 6 5.4 184.2 6.6X
+Both columns - ORC Vectorized 456 504 39 34.5 29.0 42.1X
+Both columns - ORC MR 2594 2597 4 6.1 164.9 7.4X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 7966 12723 2892 1.3 759.7 1.0X
-SQL Json 9897 10008 157 1.1 943.9 0.8X
-SQL Parquet Vectorized: DataPageV1 1176 1264 125 8.9 112.1 6.8X
-SQL Parquet Vectorized: DataPageV2 2224 2326 144 4.7 212.1 3.6X
-SQL Parquet MR: DataPageV1 3431 3483 73 3.1 327.2 2.3X
-SQL Parquet MR: DataPageV2 3845 4043 280 2.7 366.7 2.1X
-ParquetReader Vectorized: DataPageV1 1055 1056 2 9.9 100.6 7.6X
-ParquetReader Vectorized: DataPageV2 2093 2119 37 5.0 199.6 3.8X
-SQL ORC Vectorized 1129 1217 125 9.3 107.7 7.1X
-SQL ORC MR 2931 2982 72 3.6 279.5 2.7X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 9165 9257 130 1.1 874.1 1.0X
+SQL Json 10230 10234 7 1.0 975.6 0.9X
+SQL Parquet Vectorized: DataPageV1 1275 1315 56 8.2 121.6 7.2X
+SQL Parquet Vectorized: DataPageV2 2406 2407 0 4.4 229.5 3.8X
+SQL Parquet MR: DataPageV1 4005 4009 7 2.6 381.9 2.3X
+SQL Parquet MR: DataPageV2 4358 4366 12 2.4 415.6 2.1X
+ParquetReader Vectorized: DataPageV1 985 995 13 10.6 94.0 9.3X
+ParquetReader Vectorized: DataPageV2 2039 2061 32 5.1 194.4 4.5X
+SQL ORC Vectorized 1048 1072 34 10.0 99.9 8.7X
+SQL ORC MR 3179 3196 24 3.3 303.2 2.9X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 6338 6508 240 1.7 604.4 1.0X
-SQL Json 7149 7247 138 1.5 681.8 0.9X
-SQL Parquet Vectorized: DataPageV1 937 984 45 11.2 89.3 6.8X
-SQL Parquet Vectorized: DataPageV2 1582 1608 37 6.6 150.9 4.0X
-SQL Parquet MR: DataPageV1 2525 2721 277 4.2 240.8 2.5X
-SQL Parquet MR: DataPageV2 2969 2974 7 3.5 283.1 2.1X
-ParquetReader Vectorized: DataPageV1 933 940 12 11.2 88.9 6.8X
-ParquetReader Vectorized: DataPageV2 1535 1549 20 6.8 146.4 4.1X
-SQL ORC Vectorized 1144 1204 86 9.2 109.1 5.5X
-SQL ORC MR 2816 2822 8 3.7 268.6 2.3X
-
-OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1025-azure
+SQL CSV 6749 6750 2 1.6 643.6 1.0X
+SQL Json 7960 7967 10 1.3 759.1 0.8X
+SQL Parquet Vectorized: DataPageV1 1078 1105 37 9.7 102.8 6.3X
+SQL Parquet Vectorized: DataPageV2 1939 1941 3 5.4 184.9 3.5X
+SQL Parquet MR: DataPageV1 3090 3099 13 3.4 294.7 2.2X
+SQL Parquet MR: DataPageV2 3274 3286 17 3.2 312.3 2.1X
+ParquetReader Vectorized: DataPageV1 1058 1067 13 9.9 100.9 6.4X
+ParquetReader Vectorized: DataPageV2 1847 1848 2 5.7 176.2 3.7X
+SQL ORC Vectorized 1307 1307 0 8.0 124.6 5.2X
+SQL ORC MR 3078 3122 62 3.4 293.6 2.2X
+
+OpenJDK 64-Bit Server VM 11.0.14+9-LTS on Linux 5.11.0-1025-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-SQL CSV 4443 4504 86 2.4 423.7 1.0X
-SQL Json 4528 4563 49 2.3 431.8 1.0X
-SQL Parquet Vectorized: DataPageV1 213 233 15 49.2 20.3 20.8X
-SQL Parquet Vectorized: DataPageV2 267 294 22 39.3 25.4 16.7X
-SQL Parquet MR: DataPageV1 1691 1700 13 6.2 161.2 2.6X
-SQL Parquet MR: DataPageV2 1515 1565 70 6.9 144.5 2.9X
-ParquetReader Vectorized: DataPageV1 228 231 2 46.0 21.7 19.5X
-ParquetReader Vectorized: DataPageV2 285 296 9 36.8 27.1 15.6X
-SQL ORC Vectorized 369 425 82 28.4 35.2 12.1X
-SQL ORC MR 1457 1463 9 7.2 138.9 3.0X
+SQL CSV 4818 4824 8 2.2 459.5 1.0X
+SQL Json 4853 4878 35 2.2 462.8 1.0X
+SQL Parquet Vectorized: DataPageV1 255 264 6 41.1 24.3 18.9X
+SQL Parquet Vectorized: DataPageV2 711 716 4 14.7 67.8 6.8X
Review comment:
So I removed the `if (total == 0)` check and it seems to have a big impact on performance. Before:
```
[info] String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] SQL CSV 3155 3213 83 3.3 300.9 1.0X
[info] SQL Json 3934 3982 68 2.7 375.2 0.8X
[info] SQL Parquet Vectorized: DataPageV1 149 188 83 70.3 14.2 21.2X
[info] SQL Parquet Vectorized: DataPageV2 198 235 55 53.0 18.9 15.9X
[info] SQL Parquet MR: DataPageV1 1109 1155 66 9.5 105.7 2.8X
[info] SQL Parquet MR: DataPageV2 1086 1094 12 9.7 103.5 2.9X
[info] ParquetReader Vectorized: DataPageV1 110 112 1 94.9 10.5 28.6X
[info] ParquetReader Vectorized: DataPageV2 444 547 90 23.6 42.3 7.1X
[info] SQL ORC Vectorized 960 1008 64 10.9 91.6 3.3X
[info] SQL ORC MR 1374 1403 41 7.6 131.0 2.3X
```
After:
```
[info] String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] SQL CSV 3719 4191 667 2.8 354.7 1.0X
[info] SQL Json 4774 7948 1303 2.2 455.3 0.8X
[info] SQL Parquet Vectorized: DataPageV1 172 268 143 61.0 16.4 21.6X
[info] SQL Parquet Vectorized: DataPageV2 218 302 121 48.2 20.8 17.1X
[info] SQL Parquet MR: DataPageV1 1030 1075 64 10.2 98.2 3.6X
[info] SQL Parquet MR: DataPageV2 1919 2783 1222 5.5 183.0 1.9X
[info] ParquetReader Vectorized: DataPageV1 123 228 149 85.3 11.7 30.2X
[info] ParquetReader Vectorized: DataPageV2 169 176 8 62.2 16.1 22.0X
[info] SQL ORC Vectorized 269 284 9 39.0 25.7 13.8X
[info] SQL ORC MR 1006 1016 14 10.4 96.0 3.7X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] parthchandra commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
parthchandra commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r821106598
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import static org.apache.spark.sql.types.DataTypes.IntegerType;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+
+/**
+ * An implementation of the Parquet DELTA_LENGTH_BYTE_ARRAY decoder that supports the vectorized
+ * interface.
+ */
+public class VectorizedDeltaLengthByteArrayReader extends VectorizedReaderBase implements
+ VectorizedValuesReader {
+
+ private final VectorizedDeltaBinaryPackedReader lengthReader;
+ private ByteBufferInputStream in;
+ private WritableColumnVector lengthsVector;
+ private int currentRow = 0;
+
+ VectorizedDeltaLengthByteArrayReader() {
+ lengthReader = new VectorizedDeltaBinaryPackedReader();
+ }
+
+ @Override
+ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
+ lengthsVector = new OnHeapColumnVector(valueCount, IntegerType);
+ lengthReader.initFromPage(valueCount, in);
+ lengthReader.readIntegers(lengthReader.getTotalValueCount(), lengthsVector, 0);
+ this.in = in.remainingStream();
+ }
+
+ @Override
+ public void readBinary(int total, WritableColumnVector c, int rowId) {
+ ByteBuffer buffer;
+ ByteBufferOutputWriter outputWriter = ByteBufferOutputWriter::writeArrayByteBuffer;
+ int length;
+ for (int i = 0; i < total; i++) {
+ length = lengthsVector.getInt(rowId + i);
+ try {
+ buffer = in.slice(length);
+ } catch (EOFException e) {
+ throw new ParquetDecodingException("Failed to read " + length + " bytes");
+ }
+ outputWriter.write(c, rowId + i, buffer, length);
+ }
+ currentRow += total;
+ }
+
+ @Override
+ public void skipBinary(int total) {
+ if (total == 0) {
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r828483576
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java
##########
@@ -443,6 +444,8 @@ public UTF8String getUTF8String(int rowId) {
}
}
+ public abstract ByteBuffer getBytesUnsafe(int rowId, int count);
Review comment:
Yea I think we can use `getByteBuffer`. the "unsafe" here is a bit confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org