You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2025/08/11 20:24:31 UTC
[PR] Spark: Encapsulate parquet objects for Comet [iceberg]
parthchandra opened a new pull request, #13786:
URL: https://github.com/apache/iceberg/pull/13786
[DO NOT MERGE]
Fixes issues in https://github.com/apache/iceberg/pull/13378. This makes all the remaining changes to Iceberg to address issues with Comet/Parquet shading. The corresponding changes to Comet are already merged into main. (https://github.com/apache/datafusion-comet/pull/2078).
Note that this cannot be merged until Comet does a release with the required changes.
This PR also removes the commit to enable Comet execution (moving it to a follow up PR: cc @hsiang-c).
This PR also makes the corresponding changes for Spark 3.4, and Spark 4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2389365662
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ String logicalTypeName = null;
+ Map<String, String> logicalTypeParams = Maps.newHashMap();
+ LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+
+ if (logicalType != null) {
+ logicalTypeName = logicalType.getClass().getSimpleName();
+
+ // Handle specific logical types
+ if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =
+ (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("precision", String.valueOf(decimal.getPrecision()));
+ logicalTypeParams.put("scale", String.valueOf(decimal.getScale()));
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =
+ (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(timestamp.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", timestamp.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =
+ (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(time.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", time.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =
+ (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isSigned", String.valueOf(intType.isSigned()));
+ logicalTypeParams.put("bitWidth", String.valueOf(intType.getBitWidth()));
+ }
+ }
+
+ int id = -1;
+ Type type = descriptor.getPrimitiveType();
+ if (type != null && type.getId() != null) {
+ id = type.getId().intValue();
+ }
+
+ return new ParquetColumnSpec(
+ id,
+ path,
+ physicalType,
+ typeLength,
+ isRepeated,
+ descriptor.getMaxDefinitionLevel(),
+ descriptor.getMaxRepetitionLevel(),
+ logicalTypeName,
+ logicalTypeParams);
+ }
+
+ public static ColumnDescriptor buildColumnDescriptor(ParquetColumnSpec columnSpec) {
+ PrimitiveType.PrimitiveTypeName primType =
+ PrimitiveType.PrimitiveTypeName.valueOf(columnSpec.getPhysicalType());
+
+ Type.Repetition repetition;
+ if (columnSpec.getMaxRepetitionLevel() > 0) {
+ repetition = Type.Repetition.REPEATED;
+ } else if (columnSpec.getMaxDefinitionLevel() > 0) {
+ repetition = Type.Repetition.OPTIONAL;
+ } else {
+ repetition = Type.Repetition.REQUIRED;
+ }
+
+ String name = columnSpec.getPath()[columnSpec.getPath().length - 1];
+ // Reconstruct the logical type from parameters
+ LogicalTypeAnnotation logicalType = null;
+ if (columnSpec.getLogicalTypeName() != null) {
+ logicalType =
+ reconstructLogicalType(
+ columnSpec.getLogicalTypeName(), columnSpec.getLogicalTypeParams());
+ }
+
+ PrimitiveType primitiveType;
+ if (primType == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+ primitiveType =
+ org.apache.parquet.schema.Types.primitive(primType, repetition)
+ .length(columnSpec.getTypeLength())
+ .as(logicalType)
+ .id(columnSpec.getFieldId())
+ .named(name);
+ } else {
+ primitiveType =
+ Types.primitive(primType, repetition)
+ .as(logicalType)
+ .id(columnSpec.getFieldId())
+ .named(name);
+ }
+
+ return new ColumnDescriptor(
+ columnSpec.getPath(),
+ primitiveType,
+ columnSpec.getMaxRepetitionLevel(),
+ columnSpec.getMaxDefinitionLevel());
+ }
+
+ private static LogicalTypeAnnotation reconstructLogicalType(
+ String logicalTypeName, java.util.Map<String, String> params) {
+
+ switch (logicalTypeName) {
+ // MAP
+ case "MapLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.mapType();
+
+ // LIST
+ case "ListLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.listType();
+
+ // STRING
+ case "StringLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.stringType();
+
+ // MAP_KEY_VALUE
+ case "MapKeyValueLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.MapKeyValueTypeAnnotation.getInstance();
+
+ // ENUM
+ case "EnumLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.enumType();
+
+ // DECIMAL
+ case "DecimalLogicalTypeAnnotation":
+ if (!params.containsKey("scale") || !params.containsKey("precision")) {
+ throw new IllegalArgumentException(
+ "Missing required parameters for DecimalLogicalTypeAnnotation: " + params);
+ }
+ int scale = Integer.parseInt(params.get("scale"));
Review Comment:
It makes the code very verbose, but I've added the additional validation.
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
+ private final boolean[] shouldSkip;
+ private final VectorizedReader<T> model;
+ private final long totalValues;
+ private final int batchSize;
+ private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetadata;
+ private final boolean reuseContainers;
+ private int nextRowGroup = 0;
+ private long nextRowGroupStart = 0;
+ private long valuesRead = 0;
+ private T last = null;
+ private final FileReader cometReader;
+
+ FileIterator(
+ ReadConf conf,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.shouldSkip = conf.shouldSkip();
+ this.totalValues = conf.totalValues();
+ this.reuseContainers = conf.reuseContainers();
+ this.model = conf.vectorizedModel();
+ this.batchSize = conf.batchSize();
+ this.model.setBatchSize(this.batchSize);
+ this.columnChunkMetadata = conf.columnChunkMetadataForRowGroups();
+ this.cometReader =
+ newCometReader(
+ conf.file(),
+ conf.projection(),
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+
+ private FileReader newCometReader(
+ InputFile file,
+ MessageType projection,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ try {
+ ReadOptions cometOptions = ReadOptions.builder(new Configuration()).build();
+
+ FileReader fileReader =
+ new FileReader(
+ new WrappedInputFile(file),
+ cometOptions,
+ properties,
+ start,
+ length,
+ ByteBuffers.toByteArray(fileEncryptionKey),
+ ByteBuffers.toByteArray(fileAADPrefix));
+
+ List<ColumnDescriptor> columnDescriptors = projection.getColumns();
+
+ List<ParquetColumnSpec> specs = Lists.newArrayList();
+
+ for (ColumnDescriptor descriptor : columnDescriptors) {
+ ParquetColumnSpec spec = CometTypeUtils.descriptorToParquetColumnSpec(descriptor);
+ specs.add(spec);
+ }
+
+ fileReader.setRequestedSchemaFromSpecs(specs);
Review Comment:
Good catch. In the latest update, this is now accessed via a wrapper class which can now throw additional exceptions. The wrapper is now auto closeable and is explicitly closed in this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2422466675
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometIOException.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+
+/**
+ * Exception class for handling both IOException and general Exception cases when working with Comet
+ * FileReader operations. This provides a unified way to handle exceptions that may occur during
+ * reflection-based calls to Comet classes.
+ */
+public class CometIOException extends RuntimeException {
+
+ public CometIOException(String message) {
+ super(message);
+ }
+
+ public CometIOException(String message, Throwable cause) {
+ super(message, cause);
+ }
+
+ public CometIOException(Throwable cause) {
+ super(cause);
+ }
+
+ /** Creates a CometIOException from an IOException. */
+ public static CometIOException fromIOException(IOException cause) {
+ return new CometIOException("Comet IO operation failed", cause);
+ }
+
+ /** Creates a CometIOException from a general Exception. */
+ public static CometIOException fromException(Exception cause) {
Review Comment:
Seems a couple of the methods are not used. Shall we remove the no usages methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3618633972
> @parthchandra Can we target Spark 4.0 first to make review and revise easier? Meanwhile, can you add more tests for the changes?
@manuzhang appreciate your feedback and time spent on this PR. We started with just one Spark version in https://github.com/apache/iceberg/pull/13378 but that got no attention!
One reason for keeping 3.4 is because we need to get one release of Iceberg out with Spark 3.4 support because Comet still supports Spark 3.4. The proposal to remove 3.4 has already been approved so we are racing against the clock here.
If it helps, the Spark 3.4 and Spark 3.5 implementations are (nearly) identical to Spark 4.0. If the review focuses only on Spark 4.0, I can make sure the same changes are made for the other Spark version branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3412666719
@huaxingao thanks for bringing up the Spark 3.4 discussion in the mailing list. At this point, we could still move ahead with this PR as it is and if 3.4 is dropped, then the additional code here will also get dropped. Otherwise if 3.4 is dropped before we merge this then I will have to rebase and will drop 3.4 from here as well. (In that case Comet will have to provide instructions on manually patching and building Iceberg).
Meanwhile if there are any more items to address, then I will be happy to take care of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2484026894
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.parquet.ReadConf;
+import org.apache.iceberg.parquet.VectorizedReader;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
Review Comment:
nit: remove?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593021513
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -127,6 +127,7 @@
public class Parquet {
private static final Logger LOG = LoggerFactory.getLogger(Parquet.class);
+ private static final String VECTORIZED_READER_FACTORY = "read.parquet.vectorized-reader.factory";
Review Comment:
Do we have a unit test for changes in this class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2529918324
##########
spark/v4.0/build.gradle:
##########
@@ -272,6 +272,8 @@ project(":iceberg-spark:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersio
integrationImplementation project(path: ':iceberg-hive-metastore', configuration: 'testArtifacts')
integrationImplementation project(path: ":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}", configuration: 'testArtifacts')
integrationImplementation project(path: ":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}", configuration: 'testArtifacts')
+ integrationImplementation "org.apache.datafusion:comet-spark-spark3.5_2.13:${libs.versions.comet.get()}"
Review Comment:
please rebase after https://github.com/apache/iceberg/pull/14273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3392676529
@huaxingao we are still supporting Spark 3.4.0 in Comet so it will be useful if we have the Spark 3.4 changes as well. Is the removal of spark 3.4 part of 1.11.0 or after that?
Also, rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2462151507
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.hadoop.HadoopInputFile;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
Review Comment:
The main reason os to allow Comet's Parquet file reader to be available independent of engine.
`iceberg-parquet` provides Parquet reading
`iceberg-spark` provides the translation from Parquet to Spark's internal representation (ColumnarBatch in the case of vectorized readers)
`CometVectorizedParquetReader` is essentially the first part and provides a fast parallel reader. It is usable by other engines like Flink.
Because of the separation of the parquet file reading from the engine specific code in Iceberg (a Good Thing, imho),
we must create the `CometVectorizedParquetReader` in the `iceberg-parquet` module.
The classes `CometBridge` etc were created so we did not need to have a compile time dependency on Comet in iceberg-parquet. For the `CometVectorizedParquetReader` this also took care of the issue that after Parquet classes are shaded in Iceberg it is no longer possible to call any Comet method with a Parquet class in the signature. (This is the original problem this PR set out to solve).
It looks like it might be possible for us to create an extra level of indirection and move all these classes to the Spark module, but we would probably lose the capability to use Comet's faster reader for Flink.
If we are okay with that, then I can try to do it that way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra closed pull request #13786: Spark: Encapsulate parquet objects for Comet
URL: https://github.com/apache/iceberg/pull/13786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3475084816
@huaxingao PTAL. I've moved _all_ Comet classes out of `iceberg-parquet`. The code now uses a Factory method that loads a vectorized parquet reader via the Java `ServiceLoader` interface if a custom reader is configured. Comet now provides this custom reader for each Spark version supported.
This approach also eliminates the 'bridge' classes.
The only downside is that the `CometVectorizedParquetReader` and the `CometTypeUtils` are duplicated. Also, Flink will no longer automatically get Comet support. (But can be configured to use the Comet reader if the jars are in the path).
Note:
I added `VectorizedParquetReaderFactory.java` in the `iceberg-parquet` module. This interface defines two methods: `name()` and `createReader(...)` for pluggable vectorized reader implementations.
Concrete implementations are in the Spark modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3518836126
@huaxingao @pvary PTAL. Updated to not use Service loader and also broke up the createReader method to use a builder pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3688136100
> If it helps, the Spark 3.4 and Spark 3.5 implementations are (nearly) identical to Spark 4.0. If the review focuses only on Spark 4.0, I can make sure the same changes are made for the other Spark version branches.
Generally I do find it easier to review the PR if we have 1 PR focused on a single Spark version, such as for Spark 4.0. Once we get that PR merged, we can add followup PR that just copy/paste (hopefully) the same code to other Spark versions. If you look through the repo, this is what we often do as "backport" PRs.
Focusing on just the spark 4 integration might cut down the number of files to review by 1/3 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2504536430
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1537,6 +1575,20 @@ public <D> CloseableIterable<D> build() {
}
}
+ private static VectorizedParquetReaderFactory loadReaderFactory(String name) {
+ ServiceLoader<VectorizedParquetReaderFactory> loader =
Review Comment:
Here is Ryan's comment about using ServiceLoader: https://github.com/apache/iceberg/pull/12060#discussion_r1938350796
> The ServiceLoader framework is error prone and commonly broken because of Jar bundling.
Here he refers to the case when creating an uber-jar the files in the `META-INF/services/` directory are often removed/forgotten/overwrite each other which causes issues when trying to access the service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3392351218
@huaxingao any comments you would like me to address?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2412802091
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1422,16 +1428,35 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
- return new VectorizedParquetReader<>(
- file,
- schema,
- options,
- batchedReaderFunc,
- mapping,
- filter,
- reuseContainers,
- caseSensitive,
- maxRecordsPerBatch);
+ if (isComet) {
+ LOG.info("Comet vectorized reader enabled");
+ return new CometVectorizedParquetReader<>(
Review Comment:
@pvary I’ve seen your work on the reader/writer API proposal, but I haven’t kept up with all the details. We’re adding a `CometVectorizedParquetReader`, does this align with your proposal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-4093494704
This is not relevant after https://github.com/apache/iceberg/pull/15674. Thank you everyone for the review effort!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2271744511
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1352,16 +1358,35 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
- return new VectorizedParquetReader<>(
- file,
- schema,
- options,
- batchedReaderFunc,
- mapping,
- filter,
- reuseContainers,
- caseSensitive,
- maxRecordsPerBatch);
+ if (isComet) {
+ LOG.info("Comet enabled");
Review Comment:
changed
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
Review Comment:
I created a Comet specific `CometReadConf` that does not have a `ParquetFileReader` but uses the Comet `FileReader` instead and changed `CometVectorizedParquetReader` to reuse the FileReader created in `CometReadConf`.
Unfortunately, `CometVectorizedParquetReader` needs the file schema and `RowGroups` from `CometReadConf` and using Comet's `FileReader` to get this information in Iceberg violates the requirement to not use any Comet API that has a Parquet class in the signature (doing so breaks the implementation when Parquet is shaded by Iceberg). Also, the methods to get schema and row groups are also (rightly) made package-private in Comet so the change won't even compile.
I think for the moment this will have to do. We will revisit this integration once again as we move to a more native implementation in Comet.
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
+ private final boolean[] shouldSkip;
+ private final VectorizedReader<T> model;
+ private final long totalValues;
+ private final int batchSize;
+ private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetadata;
+ private final boolean reuseContainers;
+ private int nextRowGroup = 0;
+ private long nextRowGroupStart = 0;
+ private long valuesRead = 0;
+ private T last = null;
+ private final FileReader cometReader;
+
+ FileIterator(
+ ReadConf conf,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.shouldSkip = conf.shouldSkip();
+ this.totalValues = conf.totalValues();
+ this.reuseContainers = conf.reuseContainers();
+ this.model = conf.vectorizedModel();
+ this.batchSize = conf.batchSize();
+ this.model.setBatchSize(this.batchSize);
+ this.columnChunkMetadata = conf.columnChunkMetadataForRowGroups();
+ this.cometReader =
+ newCometReader(
+ conf.file(),
+ conf.projection(),
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+
+ private FileReader newCometReader(
+ InputFile file,
+ MessageType projection,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ try {
+ ReadOptions cometOptions = ReadOptions.builder(new Configuration()).build();
+
+ FileReader fileReader =
+ new FileReader(
+ new WrappedInputFile(file),
+ cometOptions,
+ properties,
+ start,
+ length,
+ ByteBuffers.toByteArray(fileEncryptionKey),
+ ByteBuffers.toByteArray(fileAADPrefix));
+
+ List<ColumnDescriptor> columnDescriptors = projection.getColumns();
+
+ List<ParquetColumnSpec> specs = Lists.newArrayList();
+
+ for (ColumnDescriptor descriptor : columnDescriptors) {
+ ParquetColumnSpec spec = CometTypeUtils.descriptorToParquetColumnSpec(descriptor);
+ specs.add(spec);
+ }
+
+ fileReader.setRequestedSchemaFromSpecs(specs);
+ return fileReader;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to open Parquet file: " + file.location(), e);
+ }
+ }
+
+ @Override
+ public boolean hasNext() {
+ return valuesRead < totalValues;
+ }
+
+ @Override
+ public T next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ if (valuesRead >= nextRowGroupStart) {
+ advance();
+ }
+
+ // batchSize is an integer, so casting to integer is safe
+ int numValuesToRead = (int) Math.min(nextRowGroupStart - valuesRead, batchSize);
+ if (reuseContainers) {
+ this.last = model.read(last, numValuesToRead);
+ } else {
+ this.last = model.read(null, numValuesToRead);
+ }
+ valuesRead += numValuesToRead;
+
+ return last;
+ }
+
+ private void advance() {
+ while (shouldSkip[nextRowGroup]) {
+ nextRowGroup += 1;
+ cometReader.skipNextRowGroup();
+ }
+ RowGroupReader pages;
+ try {
+ pages = cometReader.readNextRowGroup();
Review Comment:
In this case, `advance` is called only by `next` which is implementing the `iterator` interface and which explicitly checks for `hasNext` so `readNextRowGroup` always returns a not null value.
This could be done better, but I'll leave this as it is because -
i) it is not really clear what the behavior will be if `advance` silently does not do anything.
ii) the code (and behavior) will be consistent with the implementation in `VectorizedParquetReader`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "hsiang-c (via GitHub)" <gi...@apache.org>.
hsiang-c commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2268062833
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
+ private final boolean[] shouldSkip;
+ private final VectorizedReader<T> model;
+ private final long totalValues;
+ private final int batchSize;
+ private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetadata;
+ private final boolean reuseContainers;
+ private int nextRowGroup = 0;
+ private long nextRowGroupStart = 0;
+ private long valuesRead = 0;
+ private T last = null;
+ private final FileReader cometReader;
+
+ FileIterator(
+ ReadConf conf,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.shouldSkip = conf.shouldSkip();
+ this.totalValues = conf.totalValues();
+ this.reuseContainers = conf.reuseContainers();
+ this.model = conf.vectorizedModel();
+ this.batchSize = conf.batchSize();
+ this.model.setBatchSize(this.batchSize);
+ this.columnChunkMetadata = conf.columnChunkMetadataForRowGroups();
+ this.cometReader =
+ newCometReader(
+ conf.file(),
+ conf.projection(),
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+
+ private FileReader newCometReader(
+ InputFile file,
+ MessageType projection,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ try {
+ ReadOptions cometOptions = ReadOptions.builder(new Configuration()).build();
+
+ FileReader fileReader =
+ new FileReader(
+ new WrappedInputFile(file),
+ cometOptions,
+ properties,
+ start,
+ length,
+ ByteBuffers.toByteArray(fileEncryptionKey),
+ ByteBuffers.toByteArray(fileAADPrefix));
+
+ List<ColumnDescriptor> columnDescriptors = projection.getColumns();
+
+ List<ParquetColumnSpec> specs = Lists.newArrayList();
+
+ for (ColumnDescriptor descriptor : columnDescriptors) {
+ ParquetColumnSpec spec = CometTypeUtils.descriptorToParquetColumnSpec(descriptor);
+ specs.add(spec);
+ }
+
+ fileReader.setRequestedSchemaFromSpecs(specs);
+ return fileReader;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to open Parquet file: " + file.location(), e);
+ }
+ }
+
+ @Override
+ public boolean hasNext() {
+ return valuesRead < totalValues;
+ }
+
+ @Override
+ public T next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ if (valuesRead >= nextRowGroupStart) {
+ advance();
+ }
+
+ // batchSize is an integer, so casting to integer is safe
+ int numValuesToRead = (int) Math.min(nextRowGroupStart - valuesRead, batchSize);
+ if (reuseContainers) {
+ this.last = model.read(last, numValuesToRead);
+ } else {
+ this.last = model.read(null, numValuesToRead);
+ }
+ valuesRead += numValuesToRead;
+
+ return last;
+ }
+
+ private void advance() {
+ while (shouldSkip[nextRowGroup]) {
+ nextRowGroup += 1;
+ cometReader.skipNextRowGroup();
+ }
+ RowGroupReader pages;
+ try {
+ pages = cometReader.readNextRowGroup();
Review Comment:
`readNextRowGroup` might return `null` and `setRowGroupInfo` will make use of `pages`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2484024807
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ String logicalTypeName = null;
+ Map<String, String> logicalTypeParams = Maps.newHashMap();
+ LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+
+ if (logicalType != null) {
+ logicalTypeName = logicalType.getClass().getSimpleName();
+
+ // Handle specific logical types
+ if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =
+ (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("precision", String.valueOf(decimal.getPrecision()));
+ logicalTypeParams.put("scale", String.valueOf(decimal.getScale()));
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =
+ (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(timestamp.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", timestamp.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =
+ (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(time.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", time.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =
+ (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isSigned", String.valueOf(intType.isSigned()));
+ logicalTypeParams.put("bitWidth", String.valueOf(intType.getBitWidth()));
+ }
+ }
+
+ int id = -1;
+ Type type = descriptor.getPrimitiveType();
+ if (type != null && type.getId() != null) {
+ id = type.getId().intValue();
+ }
+
+ return new ParquetColumnSpec(
+ id,
+ path,
+ physicalType,
+ typeLength,
+ isRepeated,
+ descriptor.getMaxDefinitionLevel(),
+ descriptor.getMaxRepetitionLevel(),
+ logicalTypeName,
+ logicalTypeParams);
+ }
+
+ public static ColumnDescriptor buildColumnDescriptor(ParquetColumnSpec columnSpec) {
+ PrimitiveType.PrimitiveTypeName primType =
+ PrimitiveType.PrimitiveTypeName.valueOf(columnSpec.getPhysicalType());
+
+ Type.Repetition repetition;
+ if (columnSpec.getMaxRepetitionLevel() > 0) {
+ repetition = Type.Repetition.REPEATED;
+ } else if (columnSpec.getMaxDefinitionLevel() > 0) {
+ repetition = Type.Repetition.OPTIONAL;
+ } else {
+ repetition = Type.Repetition.REQUIRED;
+ }
+
+ String name = columnSpec.getPath()[columnSpec.getPath().length - 1];
+ // Reconstruct the logical type from parameters
+ LogicalTypeAnnotation logicalType = null;
+ if (columnSpec.getLogicalTypeName() != null) {
+ logicalType =
+ reconstructLogicalType(
+ columnSpec.getLogicalTypeName(), columnSpec.getLogicalTypeParams());
+ }
+
+ PrimitiveType primitiveType;
+ if (primType == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+ primitiveType =
+ org.apache.parquet.schema.Types.primitive(primType, repetition)
Review Comment:
nit: `Types.primitive`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2462197957
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.hadoop.HadoopInputFile;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
Review Comment:
Okay, let me refactor this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2423136868
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
+ private final boolean[] shouldSkip;
+ private final VectorizedReader<T> model;
+ private final long totalValues;
+ private final int batchSize;
+ private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetadata;
+ private final boolean reuseContainers;
+ private int nextRowGroup = 0;
+ private long nextRowGroupStart = 0;
+ private long valuesRead = 0;
+ private T last = null;
+ private final CometBridge.FileReaderWrapper cometReader;
+
+ FileIterator(
+ ReadConf conf,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ if (!CometBridge.isCometAvailable()) {
+ throw new IllegalStateException(
+ "Comet is not available in the classpath. "
+ + "Please ensure the comet-spark jar is in the classpath.");
+ }
+
+ this.shouldSkip = conf.shouldSkip();
+ this.totalValues = conf.totalValues();
+ this.reuseContainers = conf.reuseContainers();
+ this.model = conf.vectorizedModel();
+ this.batchSize = conf.batchSize();
+ this.model.setBatchSize(this.batchSize);
+ this.columnChunkMetadata = conf.columnChunkMetadataForRowGroups();
+ this.cometReader =
+ newCometReader(
+ conf.file(),
+ conf.projection(),
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+
+ private CometBridge.FileReaderWrapper newCometReader(
+ InputFile file,
+ MessageType projection,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ CometBridge.FileReaderWrapper fileReader = null;
+ try {
+ Object cometOptions = CometBridge.createReadOptions(new Configuration());
Review Comment:
I had mirrored what was done in your original PR, but you're right - there is a better way. If InputFile is a HadoopInputFile then we can get the Hadoop conf from there which is also what is being done in `Parquet.java`.
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
Review Comment:
Removed
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometIOException.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+
+/**
+ * Exception class for handling both IOException and general Exception cases when working with Comet
+ * FileReader operations. This provides a unified way to handle exceptions that may occur during
+ * reflection-based calls to Comet classes.
+ */
+public class CometIOException extends RuntimeException {
+
+ public CometIOException(String message) {
+ super(message);
+ }
+
+ public CometIOException(String message, Throwable cause) {
+ super(message, cause);
+ }
+
+ public CometIOException(Throwable cause) {
+ super(cause);
+ }
+
+ /** Creates a CometIOException from an IOException. */
+ public static CometIOException fromIOException(IOException cause) {
+ return new CometIOException("Comet IO operation failed", cause);
+ }
+
+ /** Creates a CometIOException from a general Exception. */
+ public static CometIOException fromException(Exception cause) {
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3392569484
@parthchandra Do we have Comet customers on Spark 3.4? If not, we don't need the changes on v3.4, because 3.4 has been marked as deprecated and will be removed in next release. Also Comet version has already been updated to 0.10.1. Could you please rebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "hsiang-c (via GitHub)" <gi...@apache.org>.
hsiang-c commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2268063592
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1352,16 +1358,35 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
- return new VectorizedParquetReader<>(
- file,
- schema,
- options,
- batchedReaderFunc,
- mapping,
- filter,
- reuseContainers,
- caseSensitive,
- maxRecordsPerBatch);
+ if (isComet) {
+ LOG.info("Comet enabled");
Review Comment:
(nit) Maybe `Comet vectorized reader` enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2504620910
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1537,6 +1575,20 @@ public <D> CloseableIterable<D> build() {
}
}
+ private static VectorizedParquetReaderFactory loadReaderFactory(String name) {
+ ServiceLoader<VectorizedParquetReaderFactory> loader =
Review Comment:
The goal here was to decouple the vectorized parquet reader from the common code and `ServiceLoader` merely simplifies the dynamic loading of the service (in this case the reader). We could use some other mechanism like specifying the actual reader class as a config. This puts the onus on to the user to specify the class correctly and ensure it is on the classpath.
Let me change this to specify the class explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2504668817
##########
parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReaderFactory.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * Service Provider Interface (SPI) for creating custom vectorized Parquet readers.
+ *
+ * <p>Implementations of this interface can be loaded at runtime using Java's {@link
+ * java.util.ServiceLoader} mechanism. To register an implementation, create a file named {@code
+ * META-INF/services/org.apache.iceberg.parquet.VectorizedParquetReaderFactory} containing the fully
+ * qualified class name of the implementation.
+ *
+ * <p>This allows for pluggable vectorized reader implementations (e.g., Comet, Arrow, Velox)
+ * without requiring the core parquet module to depend on specific execution engines.
+ */
+public interface VectorizedParquetReaderFactory {
+
+ /**
+ * Returns the unique identifier for this reader factory.
+ *
+ * <p>This name is used to select the reader factory via configuration. For example, "comet" for
+ * the Comet vectorized reader.
+ *
+ * @return the unique name for this factory
+ */
+ String name();
+
+ /**
+ * Creates a vectorized parquet reader with the given configuration.
+ *
+ * @param file the input file to read
+ * @param schema the expected schema for the data
+ * @param options parquet read options
+ * @param batchedReaderFunc function to create a VectorizedReader from a MessageType
+ * @param mapping name mapping for schema evolution
+ * @param filter filter expression to apply during reading
+ * @param reuseContainers whether to reuse containers for records
+ * @param caseSensitive whether column name matching should be case-sensitive
+ * @param maxRecordsPerBatch maximum number of records per batch
+ * @param properties additional properties for reader configuration
+ * @param start optional start position for reading
+ * @param length optional length to read
+ * @param fileEncryptionKey optional encryption key for the file
+ * @param fileAADPrefix optional AAD prefix for encryption
+ * @param <T> the type of records returned by the reader
+ * @return a closeable iterable of records
+ */
+ <T> CloseableIterable<T> createReader(
Review Comment:
> This is ugly. For this many parameters we usually create a builder.
The linter said so too :). Let me put in a builder.
Also, I'd be in favor of a separate model for Comet, it makes the distinction between alternate implementations clearer, imho.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3482786386
Thanks @parthchandra for the revision. Introducing a small SPI in parquet (`VectorizedParquetReaderFactory`) and loading the Comet implementation from Spark via ServiceLoader minimizes core changes while enabling pluggable vectorized readers. Looks good to me—would love others’ thoughts.
cc @pvary @aokolnychyi @amogh-jahagirdar @nastra @stevenzwu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3379116982
@huaxingao, @shangxinli, @hsiang-c this is ready for final review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2484023816
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1289,6 +1290,20 @@ public ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> struct
throw new UnsupportedOperationException("Custom types are not yet supported");
}
+ /**
+ * @deprecated Use {@link #set(String, String)} with "read.parquet.vectorized-reader.factory" =
+ * "comet" instead
+ */
+ @Deprecated
Review Comment:
Why a newly added method is deprecated? Do we need this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3549120999
> What about something like this: [pvary@4b6f7a5](https://github.com/pvary/iceberg/commit/4b6f7a5f577afe0d5f4742e4602cc34c14c0115d)
> This builds above the proposed File Format API: #12774, #12298
@pvary I'm not sure what you're asking here. Would you like this PR to be changed to build on top of the proposed API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593000444
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1378,6 +1379,34 @@ public ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> struct
return this;
}
+ /** Convenience method to enable comet */
+ public ReadBuilder enableComet(boolean enableComet) {
Review Comment:
where is this method used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2504627458
##########
parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java:
##########
@@ -60,7 +60,7 @@ class ReadConf<T> {
private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetaDataForRowGroups;
@SuppressWarnings("unchecked")
- ReadConf(
+ public ReadConf(
Review Comment:
Yes we do. The Comet reader is now in a different module and needs to access the read conf for read information. I could make it protected and then derive a class explicitly meant for custom readers, but that seems overkill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3688128583
> The proposal to remove 3.4 has already been approved so we are racing against the clock here.
I raised the thread to remove spark 3.4 support for the upcoming 1.11 release. We could delay the removal until 1.12 based on feedback. I would like to remove it if possible and just have 1 spark 3.x version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3340834509
@shangxinli thank you for taking a look. I found that there are more changes to get around parquet shading that need to be addressed; let me address your comments along with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2529915552
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -30,6 +30,15 @@ private SparkSQLProperties() {}
// Controls which Parquet reader implementation to use
public static final String PARQUET_READER_TYPE = "spark.sql.iceberg.parquet.reader-type";
public static final ParquetReaderType PARQUET_READER_TYPE_DEFAULT = ParquetReaderType.ICEBERG;
+
+ // Controls the fully qualified class name of the vectorized Parquet reader factory
+ public static final String PARQUET_VECTORIZED_READER_FACTORY =
+ "spark.sql.iceberg.parquet.vectorized-reader.factory";
+
+ // Comet vectorized reader factory class name
+ public static final String COMET_VECTORIZED_READER_FACTORY_CLASS =
+ "org.apache.iceberg.spark.parquet.CometVectorizedParquetReaderFactory";
Review Comment:
Why does this config have inconsistent style with others?
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -30,6 +30,15 @@ private SparkSQLProperties() {}
// Controls which Parquet reader implementation to use
public static final String PARQUET_READER_TYPE = "spark.sql.iceberg.parquet.reader-type";
public static final ParquetReaderType PARQUET_READER_TYPE_DEFAULT = ParquetReaderType.ICEBERG;
+
+ // Controls the fully qualified class name of the vectorized Parquet reader factory
+ public static final String PARQUET_VECTORIZED_READER_FACTORY =
+ "spark.sql.iceberg.parquet.vectorized-reader.factory";
+
+ // Comet vectorized reader factory class name
+ public static final String COMET_VECTORIZED_READER_FACTORY_CLASS =
+ "org.apache.iceberg.spark.parquet.CometVectorizedParquetReaderFactory";
Review Comment:
Why does this config have inconsistent naming style with others?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2591374515
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -137,10 +137,24 @@ public PartitionReaderFactory createReaderFactory() {
}
private ParquetBatchReadConf parquetBatchReadConf(ParquetReaderType readerType) {
- return ImmutableParquetBatchReadConf.builder()
- .batchSize(readConf.parquetBatchSize())
- .readerType(readerType)
- .build();
+ String factoryClassName = readConf.parquetVectorizedReaderFactory();
+
+ // If no explicit factory is set and reader type is COMET, use the default Comet factory
+ if (factoryClassName == null && readerType == ParquetReaderType.COMET) {
+ factoryClassName =
+ org.apache.iceberg.spark.SparkSQLProperties.COMET_VECTORIZED_READER_FACTORY_CLASS;
Review Comment:
But you can import `SparkSQLProperties` and invoke like `SparkSQLProperties.COMET_VECTORIZED_READER_FACTORY_CLASS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593008717
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1378,6 +1379,34 @@ public ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> struct
return this;
}
+ /** Convenience method to enable comet */
+ public ReadBuilder enableComet(boolean enableComet) {
+ if (enableComet) {
+ this.properties.put(
+ VECTORIZED_READER_FACTORY,
+ "org.apache.iceberg.spark.parquet.CometVectorizedParquetReaderFactory");
Review Comment:
How to keep this consistent with that in `SparkSQLProperties`? Can we put it in a common module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593000444
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1378,6 +1379,34 @@ public ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> struct
return this;
}
+ /** Convenience method to enable comet */
+ public ReadBuilder enableComet(boolean enableComet) {
Review Comment:
where is this method used? Can we just use `vectorizedReaderFactory`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "hsiang-c (via GitHub)" <gi...@apache.org>.
hsiang-c commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2268063592
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1352,16 +1358,35 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
- return new VectorizedParquetReader<>(
- file,
- schema,
- options,
- batchedReaderFunc,
- mapping,
- filter,
- reuseContainers,
- caseSensitive,
- maxRecordsPerBatch);
+ if (isComet) {
+ LOG.info("Comet enabled");
Review Comment:
(nit) Maybe `Comet vectorized reader` enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2414483546
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1422,16 +1428,35 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
- return new VectorizedParquetReader<>(
- file,
- schema,
- options,
- batchedReaderFunc,
- mapping,
- filter,
- reuseContainers,
- caseSensitive,
- maxRecordsPerBatch);
+ if (isComet) {
+ LOG.info("Comet vectorized reader enabled");
+ return new CometVectorizedParquetReader<>(
Review Comment:
@pvary thanks for verifying this. If any change is needed in future, it will be fairly easy to move code around.
@huaxingao thank you for reviewing; there is also a follow up PR - https://github.com/apache/iceberg/pull/14062 is in draft because it requires this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "hsiang-c (via GitHub)" <gi...@apache.org>.
hsiang-c commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2268058170
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
Review Comment:
This constructor will create a `ParquetFileReader` internally, do you think we could use Comet's `FileReader`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3392556679
cc @aokolnychyi @flyrain @szehon-ho @stevenzwu Appreciate it if you could please take a look too. Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2422460327
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
+ private final boolean[] shouldSkip;
+ private final VectorizedReader<T> model;
+ private final long totalValues;
+ private final int batchSize;
+ private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetadata;
+ private final boolean reuseContainers;
+ private int nextRowGroup = 0;
+ private long nextRowGroupStart = 0;
+ private long valuesRead = 0;
+ private T last = null;
+ private final CometBridge.FileReaderWrapper cometReader;
+
+ FileIterator(
+ ReadConf conf,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ if (!CometBridge.isCometAvailable()) {
+ throw new IllegalStateException(
+ "Comet is not available in the classpath. "
+ + "Please ensure the comet-spark jar is in the classpath.");
+ }
+
+ this.shouldSkip = conf.shouldSkip();
+ this.totalValues = conf.totalValues();
+ this.reuseContainers = conf.reuseContainers();
+ this.model = conf.vectorizedModel();
+ this.batchSize = conf.batchSize();
+ this.model.setBatchSize(this.batchSize);
+ this.columnChunkMetadata = conf.columnChunkMetadataForRowGroups();
+ this.cometReader =
+ newCometReader(
+ conf.file(),
+ conf.projection(),
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+
+ private CometBridge.FileReaderWrapper newCometReader(
+ InputFile file,
+ MessageType projection,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ CometBridge.FileReaderWrapper fileReader = null;
+ try {
+ Object cometOptions = CometBridge.createReadOptions(new Configuration());
Review Comment:
I can't remember how the ReadOptions works. Is it correct to have an empty Configuration here?
Do we need to verify the ParquetReadOption we build on Comet side is the same as the one on Iceberg side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3392894066
@parthchandra @hsiang-c I remember we run Comet/Iceberg integration test in Comet repo. Is it possible to add a Comet integration test in `TestRoundTrip`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3393667594
> @parthchandra There is a discussion to remove 3.4 in 1.11.0. If we need to support 3.4 in Comet for a longer time, we need to -1 in the discussion thread.
Let me post in the discussion thread and see what the community has to say.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2414340346
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1422,16 +1428,35 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
- return new VectorizedParquetReader<>(
- file,
- schema,
- options,
- batchedReaderFunc,
- mapping,
- filter,
- reuseContainers,
- caseSensitive,
- maxRecordsPerBatch);
+ if (isComet) {
+ LOG.info("Comet vectorized reader enabled");
+ return new CometVectorizedParquetReader<>(
Review Comment:
Based on the suggestion on that PR, we will just handle Comet as an internal detail for Parquet. So it will be hidden behind the FileFormat API.
Likely the `this.isComet` will be driven by a [configuration](https://github.com/apache/iceberg/pull/12774/files#diff-27afe27a8ce3704fffd857cfbe78654bea2d79b97120165ee978cfedde57a045R77) Map for the ReadBuilder, but forget about it now. It is not much of a complexity, and we will be handling it when the time comes.
Thanks for your consideration!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3617458629
@parthchandra Can we target Spark 4.0 first to make review and revise easier? Meanwhile, can you add more tests for the changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3502960849
I will try to take a look soon.
At first glance, I see that the ServiceLoader infra is used to load the different readers. When I have suggested using that on the FileFormat API PR, Ryan and Dan highlighted that it is very hard to debug and maintain services using the ServiceLoader infra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2504546523
##########
parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java:
##########
@@ -60,7 +60,7 @@ class ReadConf<T> {
private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetaDataForRowGroups;
@SuppressWarnings("unchecked")
- ReadConf(
+ public ReadConf(
Review Comment:
do we need this public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3787527560
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3176790218
@huaxingao, @anuragmantri please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2382626294
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ String logicalTypeName = null;
+ Map<String, String> logicalTypeParams = Maps.newHashMap();
+ LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+
+ if (logicalType != null) {
+ logicalTypeName = logicalType.getClass().getSimpleName();
+
+ // Handle specific logical types
+ if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =
+ (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("precision", String.valueOf(decimal.getPrecision()));
+ logicalTypeParams.put("scale", String.valueOf(decimal.getScale()));
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =
+ (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(timestamp.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", timestamp.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =
+ (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(time.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", time.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =
+ (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isSigned", String.valueOf(intType.isSigned()));
+ logicalTypeParams.put("bitWidth", String.valueOf(intType.getBitWidth()));
+ }
+ }
+
+ int id = -1;
+ Type type = descriptor.getPrimitiveType();
+ if (type != null && type.getId() != null) {
+ id = type.getId().intValue();
+ }
+
+ return new ParquetColumnSpec(
+ id,
+ path,
+ physicalType,
+ typeLength,
+ isRepeated,
+ descriptor.getMaxDefinitionLevel(),
+ descriptor.getMaxRepetitionLevel(),
+ logicalTypeName,
+ logicalTypeParams);
+ }
+
+ public static ColumnDescriptor buildColumnDescriptor(ParquetColumnSpec columnSpec) {
+ PrimitiveType.PrimitiveTypeName primType =
+ PrimitiveType.PrimitiveTypeName.valueOf(columnSpec.getPhysicalType());
+
+ Type.Repetition repetition;
+ if (columnSpec.getMaxRepetitionLevel() > 0) {
+ repetition = Type.Repetition.REPEATED;
+ } else if (columnSpec.getMaxDefinitionLevel() > 0) {
+ repetition = Type.Repetition.OPTIONAL;
+ } else {
+ repetition = Type.Repetition.REQUIRED;
+ }
+
+ String name = columnSpec.getPath()[columnSpec.getPath().length - 1];
+ // Reconstruct the logical type from parameters
+ LogicalTypeAnnotation logicalType = null;
+ if (columnSpec.getLogicalTypeName() != null) {
+ logicalType =
+ reconstructLogicalType(
+ columnSpec.getLogicalTypeName(), columnSpec.getLogicalTypeParams());
+ }
+
+ PrimitiveType primitiveType;
+ if (primType == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+ primitiveType =
+ org.apache.parquet.schema.Types.primitive(primType, repetition)
+ .length(columnSpec.getTypeLength())
+ .as(logicalType)
+ .id(columnSpec.getFieldId())
+ .named(name);
+ } else {
+ primitiveType =
+ Types.primitive(primType, repetition)
+ .as(logicalType)
+ .id(columnSpec.getFieldId())
+ .named(name);
+ }
+
+ return new ColumnDescriptor(
+ columnSpec.getPath(),
+ primitiveType,
+ columnSpec.getMaxRepetitionLevel(),
+ columnSpec.getMaxDefinitionLevel());
+ }
+
+ private static LogicalTypeAnnotation reconstructLogicalType(
+ String logicalTypeName, java.util.Map<String, String> params) {
+
+ switch (logicalTypeName) {
+ // MAP
+ case "MapLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.mapType();
+
+ // LIST
+ case "ListLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.listType();
+
+ // STRING
+ case "StringLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.stringType();
+
+ // MAP_KEY_VALUE
+ case "MapKeyValueLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.MapKeyValueTypeAnnotation.getInstance();
+
+ // ENUM
+ case "EnumLogicalTypeAnnotation":
+ return LogicalTypeAnnotation.enumType();
+
+ // DECIMAL
+ case "DecimalLogicalTypeAnnotation":
+ if (!params.containsKey("scale") || !params.containsKey("precision")) {
+ throw new IllegalArgumentException(
+ "Missing required parameters for DecimalLogicalTypeAnnotation: " + params);
+ }
+ int scale = Integer.parseInt(params.get("scale"));
Review Comment:
Should we add validation for the parameter format? for example,
If params.get("scale") returns null we will get NumberFormatException. Another case is non-numeric string.
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
+ private final boolean[] shouldSkip;
+ private final VectorizedReader<T> model;
+ private final long totalValues;
+ private final int batchSize;
+ private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetadata;
+ private final boolean reuseContainers;
+ private int nextRowGroup = 0;
+ private long nextRowGroupStart = 0;
+ private long valuesRead = 0;
+ private T last = null;
+ private final FileReader cometReader;
+
+ FileIterator(
+ ReadConf conf,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.shouldSkip = conf.shouldSkip();
+ this.totalValues = conf.totalValues();
+ this.reuseContainers = conf.reuseContainers();
+ this.model = conf.vectorizedModel();
+ this.batchSize = conf.batchSize();
+ this.model.setBatchSize(this.batchSize);
+ this.columnChunkMetadata = conf.columnChunkMetadataForRowGroups();
+ this.cometReader =
+ newCometReader(
+ conf.file(),
+ conf.projection(),
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+
+ private FileReader newCometReader(
+ InputFile file,
+ MessageType projection,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ try {
+ ReadOptions cometOptions = ReadOptions.builder(new Configuration()).build();
+
+ FileReader fileReader =
+ new FileReader(
+ new WrappedInputFile(file),
+ cometOptions,
+ properties,
+ start,
+ length,
+ ByteBuffers.toByteArray(fileEncryptionKey),
+ ByteBuffers.toByteArray(fileAADPrefix));
+
+ List<ColumnDescriptor> columnDescriptors = projection.getColumns();
+
+ List<ParquetColumnSpec> specs = Lists.newArrayList();
+
+ for (ColumnDescriptor descriptor : columnDescriptors) {
+ ParquetColumnSpec spec = CometTypeUtils.descriptorToParquetColumnSpec(descriptor);
+ specs.add(spec);
+ }
+
+ fileReader.setRequestedSchemaFromSpecs(specs);
Review Comment:
If setRequestedSchemaFromSpecs fails, the FileReader will be closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3392460313
@parthchandra Apologies for the delay. I’ve skimmed the changes and would like more time for a deeper review. Because we’re adding several classes to the Parquet module, Comet code is no longer confined to the Spark module. Given the broader scope, it would be ideal if we can get one more committer to review and sign off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2484023383
##########
build.gradle:
##########
@@ -845,6 +845,7 @@ project(':iceberg-orc') {
}
project(':iceberg-parquet') {
+
Review Comment:
nit: unnecessary change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2484027748
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1352,6 +1367,32 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
+ // Try to load custom vectorized reader factory from properties
+ String readerName = properties.get("read.parquet.vectorized-reader.factory");
Review Comment:
Shall we defined a constant for `read.parquet.vectorized-reader.factory`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2504608108
##########
parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReaderFactory.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * Service Provider Interface (SPI) for creating custom vectorized Parquet readers.
+ *
+ * <p>Implementations of this interface can be loaded at runtime using Java's {@link
+ * java.util.ServiceLoader} mechanism. To register an implementation, create a file named {@code
+ * META-INF/services/org.apache.iceberg.parquet.VectorizedParquetReaderFactory} containing the fully
+ * qualified class name of the implementation.
+ *
+ * <p>This allows for pluggable vectorized reader implementations (e.g., Comet, Arrow, Velox)
+ * without requiring the core parquet module to depend on specific execution engines.
+ */
+public interface VectorizedParquetReaderFactory {
+
+ /**
+ * Returns the unique identifier for this reader factory.
+ *
+ * <p>This name is used to select the reader factory via configuration. For example, "comet" for
+ * the Comet vectorized reader.
+ *
+ * @return the unique name for this factory
+ */
+ String name();
+
+ /**
+ * Creates a vectorized parquet reader with the given configuration.
+ *
+ * @param file the input file to read
+ * @param schema the expected schema for the data
+ * @param options parquet read options
+ * @param batchedReaderFunc function to create a VectorizedReader from a MessageType
+ * @param mapping name mapping for schema evolution
+ * @param filter filter expression to apply during reading
+ * @param reuseContainers whether to reuse containers for records
+ * @param caseSensitive whether column name matching should be case-sensitive
+ * @param maxRecordsPerBatch maximum number of records per batch
+ * @param properties additional properties for reader configuration
+ * @param start optional start position for reading
+ * @param length optional length to read
+ * @param fileEncryptionKey optional encryption key for the file
+ * @param fileAADPrefix optional AAD prefix for encryption
+ * @param <T> the type of records returned by the reader
+ * @return a closeable iterable of records
+ */
+ <T> CloseableIterable<T> createReader(
Review Comment:
This is ugly. For this many parameters we usually create a builder.
Like the WriteBuilder in this PR: https://github.com/apache/iceberg/pull/12774/files
I have been toying with the idea of creating a separate FormatModel for Comet and register it to the FormatModelRegistry. The rationale was, that we basically create an alternate reader for Parquet vectorized reads.
We could still do it if the returned batch class is different.
Like:
```
public class SparkFormatModels {
FormatModelRegistry.register(
new ParquetFormatModel<ColumnarBatch, StructType, DeleteFilter<InternalRow>>(
ColumnarBatch.class, StructType.class, VectorizedSparkParquetReaders::buildReader));
FormatModelRegistry.register(
new ParquetFormatModel<ColumnarBatch, StructType, DeleteFilter<InternalRow>>(
CometColumnarBatch.class, StructType.class, VectorizedSparkParquetReaders::buildCometReader));
}
```
And if they are compatible, then we can rely on them when getting the builder:
```
abstract class BaseBatchReader<T extends ScanTask> extends BaseReader<ColumnarBatch, T> {
[..]
ReadBuilder readBuilder =
FormatModelRegistry.readBuilder(format, config.isComet() ? CometColumnarBatch. class : ColumnarBatch.class, inputFile);
[..]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3615050552
@pvary @huaxingao I've rebased on the latest and resolved the conflicts. PTAL.
If there are no more changes to be made, can we approve and merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593988417
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -127,6 +127,7 @@
public class Parquet {
private static final Logger LOG = LoggerFactory.getLogger(Parquet.class);
+ private static final String VECTORIZED_READER_FACTORY = "read.parquet.vectorized-reader.factory";
Review Comment:
I've added a bunch of tests for this. For the Comet specific code, there is a plan to add a diff file that can be applied to change the reader for all the existing tests and we can add a ci test pipeline that verifies that nothing is broken.
##########
parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java:
##########
@@ -60,7 +60,7 @@ class ReadConf<T> {
private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetaDataForRowGroups;
@SuppressWarnings("unchecked")
- ReadConf(
+ public ReadConf(
Review Comment:
I would rather not rename an existing class. I feel it will reduce confusion on the scope of the changes.
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1537,6 +1589,27 @@ public <D> CloseableIterable<D> build() {
}
}
+ private static VectorizedParquetReaderFactory loadReaderFactory(String className) {
+ try {
+ Class<?> factoryClass = Class.forName(className);
+ if (!VectorizedParquetReaderFactory.class.isAssignableFrom(factoryClass)) {
+ LOG.warn("Class {} does not implement VectorizedParquetReaderFactory interface", className);
+ return null;
Review Comment:
We lose a little bit of helpful text in the error, but made the change.
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.parquet.ReadConf;
+import org.apache.iceberg.parquet.VectorizedReader;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private final Long start;
+ private final Long length;
+ private final ByteBuffer fileEncryptionKey;
+ private final ByteBuffer fileAADPrefix;
+
+ private CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ public static Builder builder(
+ InputFile file,
+ Schema schema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> batchedReaderFunc) {
+ return new Builder(file, schema, options, batchedReaderFunc);
+ }
+
+ public static class Builder {
+ private final InputFile file;
+ private final Schema schema;
+ private final ParquetReadOptions options;
+ private final Function<MessageType, VectorizedReader<?>> batchedReaderFunc;
+ private NameMapping nameMapping = null;
+ private Expression filter = null;
+ private boolean reuseContainers = false;
+ private boolean caseSensitive = true;
+ private int maxRecordsPerBatch = 10000;
+ private Map<String, String> properties = null;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ private Builder(
+ InputFile file,
+ Schema schema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> batchedReaderFunc) {
+ this.file = file;
+ this.schema = schema;
+ this.options = options;
+ this.batchedReaderFunc = batchedReaderFunc;
+ }
+
+ public Builder nameMapping(NameMapping mapping) {
+ this.nameMapping = mapping;
+ return this;
+ }
+
+ public Builder filter(Expression filterExpr) {
+ this.filter = filterExpr;
+ return this;
+ }
+
+ public Builder reuseContainers(boolean reuse) {
+ this.reuseContainers = reuse;
+ return this;
+ }
+
+ public Builder caseSensitive(boolean sensitive) {
+ this.caseSensitive = sensitive;
+ return this;
+ }
+
+ public Builder maxRecordsPerBatch(int maxRecords) {
+ this.maxRecordsPerBatch = maxRecords;
+ return this;
+ }
+
+ public Builder properties(Map<String, String> props) {
+ this.properties = props;
+ return this;
+ }
+
+ public Builder split(Long splitStart, Long splitLength) {
+ this.start = splitStart;
+ this.length = splitLength;
+ return this;
+ }
+
+ public Builder encryption(ByteBuffer encryptionKey, ByteBuffer aadPrefix) {
+ this.fileEncryptionKey = encryptionKey;
+ this.fileAADPrefix = aadPrefix;
+ return this;
+ }
+
+ public <T> CometVectorizedParquetReader<T> build() {
+ return new CometVectorizedParquetReader<>(
+ file,
+ schema,
+ options,
+ batchedReaderFunc,
+ nameMapping,
+ filter,
+ reuseContainers,
+ caseSensitive,
+ maxRecordsPerBatch,
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
Review Comment:
I don't think it needs to be. This is pretty much what is done in VectorizedParquetReader
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.parquet.ReadConf;
+import org.apache.iceberg.parquet.VectorizedReader;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private final Long start;
+ private final Long length;
+ private final ByteBuffer fileEncryptionKey;
+ private final ByteBuffer fileAADPrefix;
+
+ private CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ public static Builder builder(
+ InputFile file,
+ Schema schema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> batchedReaderFunc) {
+ return new Builder(file, schema, options, batchedReaderFunc);
+ }
+
+ public static class Builder {
+ private final InputFile file;
+ private final Schema schema;
+ private final ParquetReadOptions options;
+ private final Function<MessageType, VectorizedReader<?>> batchedReaderFunc;
+ private NameMapping nameMapping = null;
+ private Expression filter = null;
+ private boolean reuseContainers = false;
+ private boolean caseSensitive = true;
+ private int maxRecordsPerBatch = 10000;
+ private Map<String, String> properties = null;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ private Builder(
+ InputFile file,
+ Schema schema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> batchedReaderFunc) {
+ this.file = file;
+ this.schema = schema;
+ this.options = options;
+ this.batchedReaderFunc = batchedReaderFunc;
+ }
+
+ public Builder nameMapping(NameMapping mapping) {
+ this.nameMapping = mapping;
+ return this;
+ }
+
+ public Builder filter(Expression filterExpr) {
+ this.filter = filterExpr;
+ return this;
+ }
+
+ public Builder reuseContainers(boolean reuse) {
+ this.reuseContainers = reuse;
+ return this;
+ }
+
+ public Builder caseSensitive(boolean sensitive) {
+ this.caseSensitive = sensitive;
+ return this;
+ }
+
+ public Builder maxRecordsPerBatch(int maxRecords) {
+ this.maxRecordsPerBatch = maxRecords;
+ return this;
+ }
+
+ public Builder properties(Map<String, String> props) {
+ this.properties = props;
+ return this;
+ }
+
+ public Builder split(Long splitStart, Long splitLength) {
+ this.start = splitStart;
+ this.length = splitLength;
+ return this;
+ }
+
+ public Builder encryption(ByteBuffer encryptionKey, ByteBuffer aadPrefix) {
+ this.fileEncryptionKey = encryptionKey;
+ this.fileAADPrefix = aadPrefix;
+ return this;
+ }
+
+ public <T> CometVectorizedParquetReader<T> build() {
+ return new CometVectorizedParquetReader<>(
+ file,
+ schema,
+ options,
+ batchedReaderFunc,
+ nameMapping,
+ filter,
+ reuseContainers,
+ caseSensitive,
+ maxRecordsPerBatch,
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
Review Comment:
TBH, I don't know myself. Based it on what is being done in VectorizedParquetReader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2462001011
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.hadoop.HadoopInputFile;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
Review Comment:
@parthchandra Could you explain why `CometVectorizedParquetReader` was added to the parquet module? Should it be moved to the Spark module instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2529917685
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -30,6 +30,15 @@ private SparkSQLProperties() {}
// Controls which Parquet reader implementation to use
public static final String PARQUET_READER_TYPE = "spark.sql.iceberg.parquet.reader-type";
public static final ParquetReaderType PARQUET_READER_TYPE_DEFAULT = ParquetReaderType.ICEBERG;
+
+ // Controls the fully qualified class name of the vectorized Parquet reader factory
+ public static final String PARQUET_VECTORIZED_READER_FACTORY =
+ "spark.sql.iceberg.parquet.vectorized-reader.factory";
+
Review Comment:
nit: no space between related configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2504615667
##########
parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReaderFactory.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * Service Provider Interface (SPI) for creating custom vectorized Parquet readers.
+ *
+ * <p>Implementations of this interface can be loaded at runtime using Java's {@link
+ * java.util.ServiceLoader} mechanism. To register an implementation, create a file named {@code
+ * META-INF/services/org.apache.iceberg.parquet.VectorizedParquetReaderFactory} containing the fully
+ * qualified class name of the implementation.
+ *
+ * <p>This allows for pluggable vectorized reader implementations (e.g., Comet, Arrow, Velox)
+ * without requiring the core parquet module to depend on specific execution engines.
+ */
+public interface VectorizedParquetReaderFactory {
+
+ /**
+ * Returns the unique identifier for this reader factory.
+ *
+ * <p>This name is used to select the reader factory via configuration. For example, "comet" for
+ * the Comet vectorized reader.
+ *
+ * @return the unique name for this factory
+ */
+ String name();
+
+ /**
+ * Creates a vectorized parquet reader with the given configuration.
+ *
+ * @param file the input file to read
+ * @param schema the expected schema for the data
+ * @param options parquet read options
+ * @param batchedReaderFunc function to create a VectorizedReader from a MessageType
+ * @param mapping name mapping for schema evolution
+ * @param filter filter expression to apply during reading
+ * @param reuseContainers whether to reuse containers for records
+ * @param caseSensitive whether column name matching should be case-sensitive
+ * @param maxRecordsPerBatch maximum number of records per batch
+ * @param properties additional properties for reader configuration
+ * @param start optional start position for reading
+ * @param length optional length to read
+ * @param fileEncryptionKey optional encryption key for the file
+ * @param fileAADPrefix optional AAD prefix for encryption
+ * @param <T> the type of records returned by the reader
+ * @return a closeable iterable of records
+ */
+ <T> CloseableIterable<T> createReader(
Review Comment:
Or we could revisit adding one more parameter to the `FormatModelRegistry.readBuilder` so we don't have to hack with the different return class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3392685674
@parthchandra There is a discussion to remove 3.4 in 1.11.0. If we need to support 3.4 in Comet for a longer time, we need to -1 in the discussion thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2422459969
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
Review Comment:
nit: remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3393669259
> @parthchandra @hsiang-c I remember we run Comet/Iceberg integration test in Comet repo. Is it possible to add a Comet integration test in `TestRoundTrip`?
I believe @hsiang-c has a PR in the works for this. Longer term I was thinking we create a (manual) CI pipeline that applies a diff file to enable comet and run the all Iceberg-Spark tests. We have this running in Comet CI at the moment but it would be good to have this in Iceberg itself. We can run the pipeline only for Comet PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3392514516
Thanks @huaxingao. Who would you recommend we ask for a second review?
Note that the introduction of Comet classes becomes necessary because of the use of Comet to provide parquet file reading. If we use the Iceberg parquet file reader then we won't need to include any Comet classes in the parquet module. Also, to avoid creating a dependency on Comet, the Comet classes in the parquet module use reflection introducing the CometBridge class among others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3622378813
> The proposal to remove 3.4 has already been approved so we are racing against the clock here.
Targeting multiple Spark versions in one PR doesn't make it faster. Usually when a PR targets Spark 4.0 is merged, the back-port PR can be reviewed and merged quickly if the changes are identical. On the other hand, reviewers can get distracted if a PR targets multiple Spark versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593140947
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.parquet.ReadConf;
+import org.apache.iceberg.parquet.VectorizedReader;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private final Long start;
+ private final Long length;
+ private final ByteBuffer fileEncryptionKey;
+ private final ByteBuffer fileAADPrefix;
+
+ private CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ public static Builder builder(
+ InputFile file,
+ Schema schema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> batchedReaderFunc) {
+ return new Builder(file, schema, options, batchedReaderFunc);
+ }
+
+ public static class Builder {
+ private final InputFile file;
+ private final Schema schema;
+ private final ParquetReadOptions options;
+ private final Function<MessageType, VectorizedReader<?>> batchedReaderFunc;
+ private NameMapping nameMapping = null;
+ private Expression filter = null;
+ private boolean reuseContainers = false;
+ private boolean caseSensitive = true;
+ private int maxRecordsPerBatch = 10000;
+ private Map<String, String> properties = null;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ private Builder(
+ InputFile file,
+ Schema schema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> batchedReaderFunc) {
+ this.file = file;
+ this.schema = schema;
+ this.options = options;
+ this.batchedReaderFunc = batchedReaderFunc;
+ }
+
+ public Builder nameMapping(NameMapping mapping) {
+ this.nameMapping = mapping;
+ return this;
+ }
+
+ public Builder filter(Expression filterExpr) {
+ this.filter = filterExpr;
+ return this;
+ }
+
+ public Builder reuseContainers(boolean reuse) {
+ this.reuseContainers = reuse;
+ return this;
+ }
+
+ public Builder caseSensitive(boolean sensitive) {
+ this.caseSensitive = sensitive;
+ return this;
+ }
+
+ public Builder maxRecordsPerBatch(int maxRecords) {
+ this.maxRecordsPerBatch = maxRecords;
+ return this;
+ }
+
+ public Builder properties(Map<String, String> props) {
+ this.properties = props;
+ return this;
+ }
+
+ public Builder split(Long splitStart, Long splitLength) {
+ this.start = splitStart;
+ this.length = splitLength;
+ return this;
+ }
+
+ public Builder encryption(ByteBuffer encryptionKey, ByteBuffer aadPrefix) {
+ this.fileEncryptionKey = encryptionKey;
+ this.fileAADPrefix = aadPrefix;
+ return this;
+ }
+
+ public <T> CometVectorizedParquetReader<T> build() {
+ return new CometVectorizedParquetReader<>(
+ file,
+ schema,
+ options,
+ batchedReaderFunc,
+ nameMapping,
+ filter,
+ reuseContainers,
+ caseSensitive,
+ maxRecordsPerBatch,
+ properties,
+ start,
+ length,
+ fileEncryptionKey,
+ fileAADPrefix);
+ }
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
Review Comment:
Is this thread safe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593103947
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -137,10 +137,24 @@ public PartitionReaderFactory createReaderFactory() {
}
private ParquetBatchReadConf parquetBatchReadConf(ParquetReaderType readerType) {
- return ImmutableParquetBatchReadConf.builder()
- .batchSize(readConf.parquetBatchSize())
- .readerType(readerType)
- .build();
+ String factoryClassName = readConf.parquetVectorizedReaderFactory();
+
+ // If no explicit factory is set and reader type is COMET, use the default Comet factory
+ if (factoryClassName == null && readerType == ParquetReaderType.COMET) {
Review Comment:
This can be handled inside `readConf.parquetVectorizedReaderFactory()` and moved to right before the check `if (factoryClassName != null) {`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2539815504
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -30,6 +30,15 @@ private SparkSQLProperties() {}
// Controls which Parquet reader implementation to use
public static final String PARQUET_READER_TYPE = "spark.sql.iceberg.parquet.reader-type";
public static final ParquetReaderType PARQUET_READER_TYPE_DEFAULT = ParquetReaderType.ICEBERG;
+
+ // Controls the fully qualified class name of the vectorized Parquet reader factory
+ public static final String PARQUET_VECTORIZED_READER_FACTORY =
+ "spark.sql.iceberg.parquet.vectorized-reader.factory";
+
Review Comment:
fixed
##########
spark/v4.0/build.gradle:
##########
@@ -272,6 +272,8 @@ project(":iceberg-spark:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersio
integrationImplementation project(path: ':iceberg-hive-metastore', configuration: 'testArtifacts')
integrationImplementation project(path: ":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}", configuration: 'testArtifacts')
integrationImplementation project(path: ":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}", configuration: 'testArtifacts')
+ integrationImplementation "org.apache.datafusion:comet-spark-spark3.5_2.13:${libs.versions.comet.get()}"
Review Comment:
Corrected (This is a new line so rebasing didn't address this)
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -137,10 +137,24 @@ public PartitionReaderFactory createReaderFactory() {
}
private ParquetBatchReadConf parquetBatchReadConf(ParquetReaderType readerType) {
- return ImmutableParquetBatchReadConf.builder()
- .batchSize(readConf.parquetBatchSize())
- .readerType(readerType)
- .build();
+ String factoryClassName = readConf.parquetVectorizedReaderFactory();
+
+ // If no explicit factory is set and reader type is COMET, use the default Comet factory
+ if (factoryClassName == null && readerType == ParquetReaderType.COMET) {
+ factoryClassName =
+ org.apache.iceberg.spark.SparkSQLProperties.COMET_VECTORIZED_READER_FACTORY_CLASS;
Review Comment:
Fails checkstyle -
```
Task :iceberg-spark:iceberg-spark-4.0_2.13:checkstyleMain FAILED
[ant:checkstyle] [ERROR] /iceberg/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:21:58: Using a static member import should be avoided - org.apache.iceberg.spark.SparkSQLProperties.COMET_VECTORIZED_READER_FACTORY_CLASS. [AvoidStaticImport]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3379118977
cc @anuragmantri
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2281086680
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ // ToDo: extract this into a Util method
Review Comment:
nit: remove this line?
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ // ToDo: extract this into a Util method
+ String logicalTypeName = null;
+ Map<String, String> logicalTypeParams = Maps.newHashMap();
+ LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+
+ if (logicalType != null) {
+ logicalTypeName = logicalType.getClass().getSimpleName();
+
+ // Handle specific logical types
+ if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =
+ (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("precision", String.valueOf(decimal.getPrecision()));
+ logicalTypeParams.put("scale", String.valueOf(decimal.getScale()));
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =
+ (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(timestamp.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", timestamp.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =
+ (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(time.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", time.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =
+ (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isSigned", String.valueOf(intType.isSigned()));
+ logicalTypeParams.put("bitWidth", String.valueOf(intType.getBitWidth()));
+ }
+ }
+
+ return new ParquetColumnSpec(
+ 1, // ToDo: pass in the correct id
Review Comment:
fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2283560481
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ // ToDo: extract this into a Util method
+ String logicalTypeName = null;
+ Map<String, String> logicalTypeParams = Maps.newHashMap();
+ LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+
+ if (logicalType != null) {
+ logicalTypeName = logicalType.getClass().getSimpleName();
+
+ // Handle specific logical types
+ if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =
+ (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("precision", String.valueOf(decimal.getPrecision()));
+ logicalTypeParams.put("scale", String.valueOf(decimal.getScale()));
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =
+ (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(timestamp.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", timestamp.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =
+ (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(time.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", time.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =
+ (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isSigned", String.valueOf(intType.isSigned()));
+ logicalTypeParams.put("bitWidth", String.valueOf(intType.getBitWidth()));
+ }
+ }
+
+ return new ParquetColumnSpec(
+ 1, // ToDo: pass in the correct id
Review Comment:
Done. Thanks for pointing this out!
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ // ToDo: extract this into a Util method
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2487266122
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1289,6 +1290,20 @@ public ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> struct
throw new UnsupportedOperationException("Custom types are not yet supported");
}
+ /**
+ * @deprecated Use {@link #set(String, String)} with "read.parquet.vectorized-reader.factory" =
+ * "comet" instead
+ */
+ @Deprecated
Review Comment:
Sorry, I got confused between all these revisions and thought this was already part of the API! Updated the comment.
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ String logicalTypeName = null;
+ Map<String, String> logicalTypeParams = Maps.newHashMap();
+ LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+
+ if (logicalType != null) {
+ logicalTypeName = logicalType.getClass().getSimpleName();
+
+ // Handle specific logical types
+ if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =
+ (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("precision", String.valueOf(decimal.getPrecision()));
+ logicalTypeParams.put("scale", String.valueOf(decimal.getScale()));
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =
+ (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(timestamp.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", timestamp.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =
+ (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(time.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", time.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =
+ (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isSigned", String.valueOf(intType.isSigned()));
+ logicalTypeParams.put("bitWidth", String.valueOf(intType.getBitWidth()));
+ }
+ }
+
+ int id = -1;
+ Type type = descriptor.getPrimitiveType();
+ if (type != null && type.getId() != null) {
+ id = type.getId().intValue();
+ }
+
+ return new ParquetColumnSpec(
+ id,
+ path,
+ physicalType,
+ typeLength,
+ isRepeated,
+ descriptor.getMaxDefinitionLevel(),
+ descriptor.getMaxRepetitionLevel(),
+ logicalTypeName,
+ logicalTypeParams);
+ }
+
+ public static ColumnDescriptor buildColumnDescriptor(ParquetColumnSpec columnSpec) {
+ PrimitiveType.PrimitiveTypeName primType =
+ PrimitiveType.PrimitiveTypeName.valueOf(columnSpec.getPhysicalType());
+
+ Type.Repetition repetition;
+ if (columnSpec.getMaxRepetitionLevel() > 0) {
+ repetition = Type.Repetition.REPEATED;
+ } else if (columnSpec.getMaxDefinitionLevel() > 0) {
+ repetition = Type.Repetition.OPTIONAL;
+ } else {
+ repetition = Type.Repetition.REQUIRED;
+ }
+
+ String name = columnSpec.getPath()[columnSpec.getPath().length - 1];
+ // Reconstruct the logical type from parameters
+ LogicalTypeAnnotation logicalType = null;
+ if (columnSpec.getLogicalTypeName() != null) {
+ logicalType =
+ reconstructLogicalType(
+ columnSpec.getLogicalTypeName(), columnSpec.getLogicalTypeParams());
+ }
+
+ PrimitiveType primitiveType;
+ if (primType == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+ primitiveType =
+ org.apache.parquet.schema.Types.primitive(primType, repetition)
Review Comment:
Done
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1352,6 +1367,32 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
+ // Try to load custom vectorized reader factory from properties
+ String readerName = properties.get("read.parquet.vectorized-reader.factory");
Review Comment:
Done
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ String logicalTypeName = null;
+ Map<String, String> logicalTypeParams = Maps.newHashMap();
+ LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+
+ if (logicalType != null) {
+ logicalTypeName = logicalType.getClass().getSimpleName();
+
+ // Handle specific logical types
+ if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =
+ (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("precision", String.valueOf(decimal.getPrecision()));
+ logicalTypeParams.put("scale", String.valueOf(decimal.getScale()));
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =
+ (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(timestamp.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", timestamp.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =
+ (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(time.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", time.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =
+ (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isSigned", String.valueOf(intType.isSigned()));
+ logicalTypeParams.put("bitWidth", String.valueOf(intType.getBitWidth()));
+ }
+ }
+
+ int id = -1;
+ Type type = descriptor.getPrimitiveType();
+ if (type != null && type.getId() != null) {
+ id = type.getId().intValue();
+ }
+
+ return new ParquetColumnSpec(
+ id,
+ path,
+ physicalType,
+ typeLength,
+ isRepeated,
+ descriptor.getMaxDefinitionLevel(),
+ descriptor.getMaxRepetitionLevel(),
+ logicalTypeName,
+ logicalTypeParams);
+ }
+
+ public static ColumnDescriptor buildColumnDescriptor(ParquetColumnSpec columnSpec) {
+ PrimitiveType.PrimitiveTypeName primType =
+ PrimitiveType.PrimitiveTypeName.valueOf(columnSpec.getPhysicalType());
+
+ Type.Repetition repetition;
+ if (columnSpec.getMaxRepetitionLevel() > 0) {
+ repetition = Type.Repetition.REPEATED;
+ } else if (columnSpec.getMaxDefinitionLevel() > 0) {
+ repetition = Type.Repetition.OPTIONAL;
+ } else {
+ repetition = Type.Repetition.REQUIRED;
+ }
+
+ String name = columnSpec.getPath()[columnSpec.getPath().length - 1];
+ // Reconstruct the logical type from parameters
+ LogicalTypeAnnotation logicalType = null;
+ if (columnSpec.getLogicalTypeName() != null) {
+ logicalType =
+ reconstructLogicalType(
+ columnSpec.getLogicalTypeName(), columnSpec.getLogicalTypeParams());
+ }
+
+ PrimitiveType primitiveType;
+ if (primType == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+ primitiveType =
+ org.apache.parquet.schema.Types.primitive(primType, repetition)
+ .length(columnSpec.getTypeLength())
+ .as(logicalType)
+ .id(columnSpec.getFieldId())
+ .named(name);
+ } else {
+ primitiveType =
+ Types.primitive(primType, repetition)
+ .as(logicalType)
+ .id(columnSpec.getFieldId())
+ .named(name);
+ }
+
+ return new ColumnDescriptor(
+ columnSpec.getPath(),
+ primitiveType,
+ columnSpec.getMaxRepetitionLevel(),
+ columnSpec.getMaxDefinitionLevel());
+ }
+
+ private static LogicalTypeAnnotation reconstructLogicalType(
+ String logicalTypeName, java.util.Map<String, String> params) {
Review Comment:
Done
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.comet.parquet.FileReader;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.comet.parquet.ReadOptions;
+import org.apache.comet.parquet.RowGroupReader;
+import org.apache.comet.parquet.WrappedInputFile;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.parquet.ReadConf;
+import org.apache.iceberg.parquet.VectorizedReader;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
+ implements CloseableIterable<T> {
+ private final InputFile input;
+ private final ParquetReadOptions options;
+ private final Schema expectedSchema;
+ private final Function<MessageType, VectorizedReader<?>> batchReaderFunc;
+ private final Expression filter;
+ private final boolean reuseContainers;
+ private final boolean caseSensitive;
+ private final int batchSize;
+ private final NameMapping nameMapping;
+ private final Map<String, String> properties;
+ private Long start = null;
+ private Long length = null;
+ private ByteBuffer fileEncryptionKey = null;
+ private ByteBuffer fileAADPrefix = null;
+
+ public CometVectorizedParquetReader(
+ InputFile input,
+ Schema expectedSchema,
+ ParquetReadOptions options,
+ Function<MessageType, VectorizedReader<?>> readerFunc,
+ NameMapping nameMapping,
+ Expression filter,
+ boolean reuseContainers,
+ boolean caseSensitive,
+ int maxRecordsPerBatch,
+ Map<String, String> properties,
+ Long start,
+ Long length,
+ ByteBuffer fileEncryptionKey,
+ ByteBuffer fileAADPrefix) {
+ this.input = input;
+ this.expectedSchema = expectedSchema;
+ this.options = options;
+ this.batchReaderFunc = readerFunc;
+ // replace alwaysTrue with null to avoid extra work evaluating a trivial filter
+ this.filter = filter == Expressions.alwaysTrue() ? null : filter;
+ this.reuseContainers = reuseContainers;
+ this.caseSensitive = caseSensitive;
+ this.batchSize = maxRecordsPerBatch;
+ this.nameMapping = nameMapping;
+ this.properties = properties;
+ this.start = start;
+ this.length = length;
+ this.fileEncryptionKey = fileEncryptionKey;
+ this.fileAADPrefix = fileAADPrefix;
+ }
+
+ private ReadConf conf = null;
+
+ private ReadConf init() {
+ if (conf == null) {
+ ReadConf readConf =
+ new ReadConf(
+ input,
+ options,
+ expectedSchema,
+ filter,
+ null,
+ batchReaderFunc,
+ nameMapping,
+ reuseContainers,
+ caseSensitive,
+ batchSize);
+ this.conf = readConf.copy();
+ return readConf;
+ }
+ return conf;
+ }
+
+ @Override
+ public CloseableIterator<T> iterator() {
+ FileIterator<T> iter =
+ new FileIterator<>(init(), properties, start, length, fileEncryptionKey, fileAADPrefix);
+ addCloseable(iter);
+ return iter;
+ }
+
+ private static class FileIterator<T> implements CloseableIterator<T> {
+ // private final ParquetFileReader reader;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2484024240
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/parquet/CometTypeUtils.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.parquet;
+
+import java.util.Map;
+import org.apache.comet.parquet.ParquetColumnSpec;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+public class CometTypeUtils {
+
+ private CometTypeUtils() {}
+
+ public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {
+
+ String[] path = descriptor.getPath();
+ PrimitiveType primitiveType = descriptor.getPrimitiveType();
+ String physicalType = primitiveType.getPrimitiveTypeName().name();
+
+ int typeLength =
+ primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
+ ? primitiveType.getTypeLength()
+ : 0;
+
+ boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;
+
+ String logicalTypeName = null;
+ Map<String, String> logicalTypeParams = Maps.newHashMap();
+ LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+
+ if (logicalType != null) {
+ logicalTypeName = logicalType.getClass().getSimpleName();
+
+ // Handle specific logical types
+ if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =
+ (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("precision", String.valueOf(decimal.getPrecision()));
+ logicalTypeParams.put("scale", String.valueOf(decimal.getScale()));
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =
+ (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(timestamp.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", timestamp.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =
+ (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isAdjustedToUTC", String.valueOf(time.isAdjustedToUTC()));
+ logicalTypeParams.put("unit", time.getUnit().name());
+ } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {
+ LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =
+ (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;
+ logicalTypeParams.put("isSigned", String.valueOf(intType.isSigned()));
+ logicalTypeParams.put("bitWidth", String.valueOf(intType.getBitWidth()));
+ }
+ }
+
+ int id = -1;
+ Type type = descriptor.getPrimitiveType();
+ if (type != null && type.getId() != null) {
+ id = type.getId().intValue();
+ }
+
+ return new ParquetColumnSpec(
+ id,
+ path,
+ physicalType,
+ typeLength,
+ isRepeated,
+ descriptor.getMaxDefinitionLevel(),
+ descriptor.getMaxRepetitionLevel(),
+ logicalTypeName,
+ logicalTypeParams);
+ }
+
+ public static ColumnDescriptor buildColumnDescriptor(ParquetColumnSpec columnSpec) {
+ PrimitiveType.PrimitiveTypeName primType =
+ PrimitiveType.PrimitiveTypeName.valueOf(columnSpec.getPhysicalType());
+
+ Type.Repetition repetition;
+ if (columnSpec.getMaxRepetitionLevel() > 0) {
+ repetition = Type.Repetition.REPEATED;
+ } else if (columnSpec.getMaxDefinitionLevel() > 0) {
+ repetition = Type.Repetition.OPTIONAL;
+ } else {
+ repetition = Type.Repetition.REQUIRED;
+ }
+
+ String name = columnSpec.getPath()[columnSpec.getPath().length - 1];
+ // Reconstruct the logical type from parameters
+ LogicalTypeAnnotation logicalType = null;
+ if (columnSpec.getLogicalTypeName() != null) {
+ logicalType =
+ reconstructLogicalType(
+ columnSpec.getLogicalTypeName(), columnSpec.getLogicalTypeParams());
+ }
+
+ PrimitiveType primitiveType;
+ if (primType == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+ primitiveType =
+ org.apache.parquet.schema.Types.primitive(primType, repetition)
+ .length(columnSpec.getTypeLength())
+ .as(logicalType)
+ .id(columnSpec.getFieldId())
+ .named(name);
+ } else {
+ primitiveType =
+ Types.primitive(primType, repetition)
+ .as(logicalType)
+ .id(columnSpec.getFieldId())
+ .named(name);
+ }
+
+ return new ColumnDescriptor(
+ columnSpec.getPath(),
+ primitiveType,
+ columnSpec.getMaxRepetitionLevel(),
+ columnSpec.getMaxDefinitionLevel());
+ }
+
+ private static LogicalTypeAnnotation reconstructLogicalType(
+ String logicalTypeName, java.util.Map<String, String> params) {
Review Comment:
nit: `java.util.Map` -> `Map`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2462195653
##########
parquet/src/main/java/org/apache/iceberg/parquet/CometVectorizedParquetReader.java:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.hadoop.HadoopInputFile;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.ByteBuffers;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.schema.MessageType;
+
+public class CometVectorizedParquetReader<T> extends CloseableGroup
Review Comment:
Thanks @parthchandra for the detailed explanation. I’d prefer we keep this in the Spark module for now. If we later have a concrete need to use Comet’s Parquet reader with Flink, we can revisit and refactor accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-4026374717
The last [PR](https://github.com/apache/iceberg/pull/15328) for the file format API changes in Spark is now merged. @parthchandra would you like to update this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#issuecomment-3533418746
What about something like this: https://github.com/pvary/iceberg/commit/4b6f7a5f577afe0d5f4742e4602cc34c14c0115d
This builds above the proposed File Format API: #12774, #12298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2529915552
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -30,6 +30,15 @@ private SparkSQLProperties() {}
// Controls which Parquet reader implementation to use
public static final String PARQUET_READER_TYPE = "spark.sql.iceberg.parquet.reader-type";
public static final ParquetReaderType PARQUET_READER_TYPE_DEFAULT = ParquetReaderType.ICEBERG;
+
+ // Controls the fully qualified class name of the vectorized Parquet reader factory
+ public static final String PARQUET_VECTORIZED_READER_FACTORY =
+ "spark.sql.iceberg.parquet.vectorized-reader.factory";
+
+ // Comet vectorized reader factory class name
+ public static final String COMET_VECTORIZED_READER_FACTORY_CLASS =
+ "org.apache.iceberg.spark.parquet.CometVectorizedParquetReaderFactory";
Review Comment:
Why does this config have inconsistent naming style with others?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2529916982
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -137,10 +137,24 @@ public PartitionReaderFactory createReaderFactory() {
}
private ParquetBatchReadConf parquetBatchReadConf(ParquetReaderType readerType) {
- return ImmutableParquetBatchReadConf.builder()
- .batchSize(readConf.parquetBatchSize())
- .readerType(readerType)
- .build();
+ String factoryClassName = readConf.parquetVectorizedReaderFactory();
+
+ // If no explicit factory is set and reader type is COMET, use the default Comet factory
+ if (factoryClassName == null && readerType == ParquetReaderType.COMET) {
+ factoryClassName =
+ org.apache.iceberg.spark.SparkSQLProperties.COMET_VECTORIZED_READER_FACTORY_CLASS;
Review Comment:
Why not import `SparkSQLProperties`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593035130
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1537,6 +1589,27 @@ public <D> CloseableIterable<D> build() {
}
}
+ private static VectorizedParquetReaderFactory loadReaderFactory(String className) {
+ try {
+ Class<?> factoryClass = Class.forName(className);
+ if (!VectorizedParquetReaderFactory.class.isAssignableFrom(factoryClass)) {
+ LOG.warn("Class {} does not implement VectorizedParquetReaderFactory interface", className);
+ return null;
Review Comment:
Rather than `return null` for multiple failure cases, how about
```java
VectorizedParquetReaderFactory factory = null;
try {
Class<?> factoryClass = Class.forName(className);
if (VectorizedParquetReaderFactory.class.isAssignableFrom(factoryClass)) {
factory = (VectorizedParquetReaderFactory) factoryClass.getDeclaredConstructor().newInstance();
} else {
// log warning
} catch (...) {
// log warning
}
return factory;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593022860
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1442,6 +1471,29 @@ public <D> CloseableIterable<D> build() {
}
if (batchedReaderFunc != null) {
+ // Try to load custom vectorized reader factory from properties
+ String readerName = properties.get(VECTORIZED_READER_FACTORY);
Review Comment:
isn't this `factoryName`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Encapsulate parquet objects for Comet [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #13786:
URL: https://github.com/apache/iceberg/pull/13786#discussion_r2593115538
##########
parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java:
##########
@@ -60,7 +60,7 @@ class ReadConf<T> {
private final List<Map<ColumnPath, ColumnChunkMetaData>> columnChunkMetaDataForRowGroups;
@SuppressWarnings("unchecked")
- ReadConf(
+ public ReadConf(
Review Comment:
If it's used outside parquet module, it would be better renamed `ParquetReadConf`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org