You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pvary (via GitHub)" <gi...@apache.org> on 2025/04/11 13:44:09 UTC
[PR] Core, Data: File Format API interfaces [iceberg]
pvary opened a new pull request, #12774:
URL: https://github.com/apache/iceberg/pull/12774
The interface part of the changes from #12298
Interfaces which have to be implemented by the File Formats:
- `ReadBuilder` - Builder for reading data from data files
- `AppenderBuilder` - Builder for writing data to data files
- `ObjectModel` - Providing ReadBuilders, and AppenderBuilders for the specific data file format and object model pair
Interfaces which are used by the actual readers/writers:
- `AppenderBuilder` - Builder for writing a file
- `DataWriterBuilder` - Builder for generating a data file
- `PositionDeleteWriterBuilder` - Builder for generating a position delete file
- `EqualityDeleteWriterBuilder` - Builder for generating an equality delete file
- No `ReadBuilder` here - the file format reader builder is reused
Implementation classes tying them together:
- `WriterBuilder` class which implements the AppenderBuilder/DataWriterBuilder/PositionDeleteWriterBuilder/EqualityDeleteWriterBuilder interfaces using the AppenderBuilder provided by the File Format itself
- `ObjectModelRegistry` which stores the available `ObjectModels` and users could request the readers (ReadBuilder) and writers (AppenderBuilder/DataWriterBuilder/PositionDeleteWriterBuilder/EqualityDeleteWriterBuilder) from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162711922
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
Review Comment:
This is the schema class used by the object model, right? I think that could be more obvious from both the name (S?) and description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2097607928
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
I tried several ways, but the information required from the engines is different for different file formats:
For Avro:
```
public ObjectModel(
String name,
BiFunction<org.apache.iceberg.Schema, Map<Integer, ?>, DatumReader<?>> readerFunction,
BiFunction<Schema, E, DatumWriter<?>> writerFunction,
BiFunction<Schema, E, DatumWriter<?>> deleteRowWriterFunction)
```
For Parquet:
```
private ObjectModel(
String name,
ReaderFunction<D> readerFunction,
BatchReaderFunction<D, F> batchReaderFunction,
WriterFunction<D, E> writerFunction,
Function<CharSequence, ?> pathTransformFunc) {
[..]
public interface ReaderFunction<D> {
ParquetValueReader<D> read(
Schema schema, MessageType messageType, Map<Integer, ?> constantFieldAccessors);
}
public interface BatchReaderFunction<D, F> {
VectorizedReader<D> read(
Schema schema,
MessageType messageType,
Map<Integer, ?> constantFieldAccessors,
F deleteFilter,
Map<String, String> config);
}
public interface WriterFunction<D, E> {
ParquetValueWriter<D> write(E engineSchema, Schema icebergSchema, MessageType messageType);
}
```
For ORC:
```
private ObjectModel(
String name,
ReaderFunction<D> readerFunction,
BatchReaderFunction<D> batchReaderFunction,
WriterFunction<E> writerFunction,
Function<CharSequence, ?> pathTransformFunc) {
[..]
public interface WriterFunction<E> {
OrcRowWriter<?> write(Schema schema, TypeDescription messageType, E nativeSchema);
}
public interface ReaderFunction<D> {
OrcRowReader<D> read(
Schema schema, TypeDescription messageType, Map<Integer, ?> constantFieldAccessors);
}
public interface BatchReaderFunction<D> {
OrcBatchReader<D> read(
Schema schema, TypeDescription messageType, Map<Integer, ?> constantFieldAccessors);
}
```
I don't see how we can push this behind a meaningful common interface.
I have tried another approach, where the `WriteBuilder`, `ReadBuilder` provides an API to return "all" of the provided configuration values, like `schema`, `projection`, `nameMapping` etc, but I decided against it because of the following reasons:
- Engine and File Format often would need shared calculations above these configuration values
- Getters on a builder is an anti-pattern
- We aim for a single transformation - keeping a single ObjectModel (or whatever we call it š) for that seems like a natural choice
Another possibility could be that we define an intermediate Object Model (maybe something like Arrow), and provide a double transformation File Format -> Arrow -> Engine, and Engine -> Arrow -> File Format. If we don't materialize the intermediate model, then we lose performance only on the double transformation. The issue with this is that it is an even bigger overhaul of the reader/write API, and I expect that there will be a serious performance hit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2097742734
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
Renamed the `ObjectModel` to `FileAccessFactory` as I find this as a best candidate for now.
Also updated a ton of javadoc, so it is easier to understand the responsibilities of the different components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2098711363
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
Review Comment:
Why should the writer need to know the content type? Shouldn't it just need a schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2100276172
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If the reader does not support encryption,
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (aad) prefix used for writing the file. If the reader
+ * does not support encryption, then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the engine-specific schema for the input data records.
Review Comment:
Went through the doc, and in most of the places changed the engine-specific to `input/output`, but kept the engine in some places, because that helps understanding the reason behind this schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444420667
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
Review Comment:
Yeah. It is not used yet, but we plan to use this for shredding which is not implemented yet.
I wanted to show what we plan in the future.
I'm ok to removing it for now f we want to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444531994
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444550814
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
Review Comment:
Outdated
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
Review Comment:
Outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444536477
##########
core/src/test/java/org/apache/iceberg/formats/TestFormatModelRegistry.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.AssertionsForInterfaceTypes.assertThat;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.util.Pair;
+import org.junit.jupiter.api.Test;
+
+class TestFormatModelRegistry {
+ /** Tests that registering the same class with the same configuration is successful. */
+ @Test
+ void testSuccessfulReRegister() {
+ // It is fine to register the same classes with the same configuration multiple times
Review Comment:
Moved
##########
core/src/test/java/org/apache/iceberg/formats/TestFormatModelRegistry.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.AssertionsForInterfaceTypes.assertThat;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.util.Pair;
+import org.junit.jupiter.api.Test;
+
+class TestFormatModelRegistry {
+ /** Tests that registering the same class with the same configuration is successful. */
+ @Test
+ void testSuccessfulReRegister() {
+ // It is fine to register the same classes with the same configuration multiple times
+ DummyFormatModel model = new DummyFormatModel(Object.class, Object.class);
+ FormatModelRegistry.register(model);
+ assertThat(FormatModelRegistry.models())
+ .containsEntry(Pair.of(FileFormat.PARQUET, Object.class), model);
+ FormatModelRegistry.register(model);
+ assertThat(FormatModelRegistry.models())
+ .containsEntry(Pair.of(FileFormat.PARQUET, Object.class), model);
+ }
+
+ /** Tests that registering the same class with the same configuration updates the registration. */
+ @Test
+ void testUpdatedRegistration() {
+ DummyFormatModel model1 = new DummyFormatModel(Object.class, Object.class);
+ DummyFormatModel model2 = new DummyFormatModel(Object.class, Object.class);
+ FormatModelRegistry.register(model1);
+ assertThat(FormatModelRegistry.models().get(Pair.of(model1.format(), model1.type())))
+ .isEqualTo(model1);
+
+ // Registering a new model with the same format and schema type should replace the old one
+ FormatModelRegistry.register(model2);
+ assertThat(FormatModelRegistry.models().get(Pair.of(model1.format(), model1.type())))
+ .isNotEqualTo(model1);
+ assertThat(FormatModelRegistry.models().get(Pair.of(model1.format(), model1.type())))
+ .isEqualTo(model2);
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2727082507
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
Review Comment:
Answered here: https://github.com/apache/iceberg/pull/12774#discussion_r2726860284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2726948339
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static synchronized void register(FormatModel<?, ?> formatModel) {
Review Comment:
The method should be called very rarely, and we also would like to put the checks for the existing keys in the synchronized block. If we do this, then we don't gain too much by leaving out the key generation out of the synchronized block, so I would keep this as it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2768937825
##########
core/src/main/java/org/apache/iceberg/deletes/PositionDeleteWriter.java:
##########
@@ -60,7 +60,7 @@ public class PositionDeleteWriter<T> implements FileWriter<PositionDelete<T>, De
private DeleteFile deleteFile = null;
public PositionDeleteWriter(
- FileAppender<StructLike> appender,
+ FileAppender<PositionDelete<T>> appender,
Review Comment:
We should doubleācheck this. While it is not a binary breaking change due to type erasure, it could still be a sourceālevel breaking change if external users are using this class with concrete StructLike writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2776210561
##########
core/src/main/java/org/apache/iceberg/formats/BaseFormatModel.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.Map;
+import org.apache.iceberg.Schema;
+
+/**
+ * Base implementation of {@link FormatModel} that provides common functionality for format models.
+ *
+ * <p>This abstract class serves as a foundation for creating format-specific models that handle
+ * reading and writing data in various file formats.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ * @param <W> the writer type produced by the writer function
+ * @param <R> the reader type produced by the reader function
+ * @param <F> the file schema type used by the underlying file format
+ */
+public abstract class BaseFormatModel<D, S, W, R, F> implements FormatModel<D, S> {
+ private final Class<? extends D> type;
+ private final Class<S> schemaType;
+ private final WriterFunction<W, S, F> writerFunction;
+ private final ReaderFunction<R, S, F> readerFunction;
+
+ /**
+ * Constructs a new BaseFormatModel with the specified configuration.
+ *
+ * @param type the row type class for the object model implementation processed by this factory.
+ * @param schemaType the schema type class for the object model implementation processed by this
+ * factory.
+ * @param writerFunction the function used to create writers for this format
+ * @param readerFunction the function used to create readers for this format
+ */
+ protected BaseFormatModel(
+ Class<? extends D> type,
+ Class<S> schemaType,
+ WriterFunction<W, S, F> writerFunction,
+ ReaderFunction<R, S, F> readerFunction) {
+ this.type = type;
+ this.schemaType = schemaType;
+ this.writerFunction = writerFunction;
+ this.readerFunction = readerFunction;
+ }
+
+ @Override
+ public Class<? extends D> type() {
+ return type;
+ }
+
+ @Override
+ public Class<S> schemaType() {
+ return schemaType;
+ }
+
+ /**
+ * Returns the writer function used to create writers for this format.
+ *
+ * @return the writer function
+ */
+ protected WriterFunction<W, S, F> writerFunction() {
+ return writerFunction;
+ }
+
+ /**
+ * Returns the reader function used to create readers for this format.
+ *
+ * @return the reader function
+ */
+ protected ReaderFunction<R, S, F> readerFunction() {
+ return readerFunction;
+ }
+
+ /**
+ * A functional interface for creating writers that can write data in a specific format.
+ *
+ * @param <W> the writer type to be created
+ * @param <S> the type of the schema for the input data
+ * @param <F> the file schema type used by the underlying file format
+ */
+ @FunctionalInterface
+ public interface WriterFunction<W, S, F> {
+ /**
+ * Creates a writer for the given schemas.
+ *
+ * @param icebergSchema the Iceberg schema defining the table structure
+ * @param fileSchema the file format specific target schema for the output files
Review Comment:
I think this should state how the file schema is derived because it is always a direct translation from the Iceberg schema and the structure and names match.
For the engine schema, I think mentioning that it is engine-specific is the right thing to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2447577328
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder meta(String property, String value);
+
+ /**
+ * Sets multiple file metadata properties in the created file.
+ *
+ * @param properties file metadata properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /**
+ * Based on the target file content the generated {@link FileAppender} needs different
+ * configuration.
+ */
+ WriteBuilder content(FileContent content);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ WriteBuilder metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ WriteBuilder overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ WriteBuilder withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the reader
+ * does not support encryption, then an exception should be thrown.
+ */
+ WriteBuilder withAADPrefix(ByteBuffer aadPrefix);
+
+ /** Finalizes the configuration and builds the {@link FileAppender}. */
+ <D> FileAppender<D> build() throws IOException;
Review Comment:
The `FormatModelRegistry` no longer exposes `WriteBuilder` or even the registered `FormatModel` instances. While users can manually create their own `FormatModel` and obtain a `WriteBuilder` from it, the API does not provide any built-in tools or support for this process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2451167439
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
Review Comment:
I agree with @stevenzwu generic types would be better. There are several benefits here:
1. Better readability.
2. More consistent with other apis such as `DataFileWriterBuilder`.
> In the future, the implementations may be updated. Also if we add a new file format support in the future, the new implementation can be properly typed.
+1 for this. One of the goals of this effort is to make adopting new file formats easier. Generic types would benefit other file formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2509973607
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.InputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Return the row type class for the object model implementation processed by this factory.
+ *
+ * <p>The model types act as a contract specifying the expected data structures for both reading
+ * (converting file formats into output objects) and writing (converting input objects into file
+ * formats). This ensures proper integration between Iceberg's storage layer and processing
+ * engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return the type of the data structures handled by this model implementation
+ */
+ Class<D> type();
+
+ /**
+ * Return the schema type class for the object model implementation processed by this factory.
+ *
+ * @return the type of the schema for the data structures handled by this model implementation
+ */
+ Class<S> schemaType();
+
+ /**
+ * Creates a writer builder for data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder
+ */
+ WriteBuilder<D, S> writeBuilder(EncryptedOutputFile outputFile);
Review Comment:
@ggershinsky: The File Format API is designed to handle data and delete file writes. From what I understand, in this case native Parquet encryption is being used, which means the input is a `NativeEncryptingOutputFile`.
Given that, the File Format API should only accept `OutputFile` as inputānot `EncryptingOutputFile`āsince data and delete files are always natively encrypted.
What I donāt understand is why `FileWriterFactory` expects `EncryptedOutputFile`:
```
public interface FileWriterFactory<T> {
DataWriter<T> newDataWriter(EncryptedOutputFile file, PartitionSpec spec, StructLike partition);
EqualityDeleteWriter<T> newEqualityDeleteWriter(EncryptedOutputFile file, PartitionSpec spec, StructLike partition);
PositionDeleteWriter<T> newPositionDeleteWriter(EncryptedOutputFile file, PartitionSpec spec, StructLike partition);
}
```
All of these should have either `NativeEncryptingOutputFile` or `OutputFile` as an input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2258456133
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
Review Comment:
Removed this parameter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254931584
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
Review Comment:
This should use `Preconditions.checkArgument` since it throws `IllegalArgumentException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2340494685
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B, S>, S> {
Review Comment:
My understanding is that this was what others required. I don't have a strong opinion, and happy to move in any direction the community decides
##########
core/src/main/java/org/apache/iceberg/io/FormatModel.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.util.function.Function;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.FormatModelRegistry;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
+ *
+ * <p>The model names (for example: "generic", "spark", "spark-vectorized", "flink", "arrow") act
+ * as a contract specifying the expected data structures for both reading (converting file formats
+ * into output objects) and writing (converting input objects into file formats). This ensures
+ * proper integration between Iceberg's storage layer and processing engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return string identifier for this model implementation
+ */
+ Class<D> type();
+
+ /**
+ * Creates a writer builder for standard data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory. The builder allows for configuration of
+ * various writing aspects like schema, metrics collection, compression, encryption.
+ *
+ * <p>The builder follows the fluent pattern for configuring writer properties and ultimately
+ * creates a {@link FileAppender} for writing the files.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder
+ */
+ WriteBuilder<D, S> writeBuilder(OutputFile outputFile);
+
+ /**
+ * Creates a function that converts {@link PositionDelete} objects into the format-specific
+ * objects which is used for writing position deletes
+ *
+ * @param schema of the position delete row content
+ * @return a function that converts {@link PositionDelete} objects into the format-specific
+ * objects.
+ */
+ Function<PositionDelete<D>, D> positionDeleteConverter(Schema schema);
+
+ /**
+ * Creates a file reader builder for the specified input file.
+ *
+ * <p>The returned {@link ReadBuilder} configures and creates a reader that converts data from the
Review Comment:
Removed most of the doc. Kept only a minimal part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2100282469
##########
data/src/main/java/org/apache/iceberg/data/FileAccessFactoryRegistry.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAccessFactory;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified file access
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>File access factories are registered through {@link
+ * #registerFileAccessFactory(FileAccessFactory)} and used for creating readers and writers. Read
+ * builders are returned directly from the factory, while write builders may be wrapped in
+ * specialized content file writer implementations depending on the requested operation type.
+ */
+public final class FileAccessFactoryRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FileAccessFactoryRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, FileAccessFactory<?>> FILE_ACCESS_FACTORIES =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers a file access factory with this registry.
+ *
+ * <p>File access factories create readers and writers for specific combinations of file formats
+ * (Parquet, ORC, Avro) and object models ("generic", "spark", "flink", etc.). Registering custom
+ * factories allows integration of new data processing engines for the supported file formats with
+ * Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param fileAccessFactory the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FileAccessFactory#format()} and {@link FileAccessFactory#objectModeName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerFileAccessFactory(FileAccessFactory<?> fileAccessFactory) {
+ Key key = new Key(fileAccessFactory.format(), fileAccessFactory.objectModeName());
+ if (FILE_ACCESS_FACTORIES.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model %s clashes with %s. Both serves %s",
+ fileAccessFactory.getClass(), FILE_ACCESS_FACTORIES.get(key), key));
+ }
+
+ FILE_ACCESS_FACTORIES.put(key, fileAccessFactory);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FileAccessFactoryRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to engine-specific objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param objectModelName identifier for the engine-specific data representation (generic, spark,
+ * flink, etc.)
+ * @param inputFile source file to read data from
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static ReadBuilder<?> readBuilder(
+ FileFormat format, String objectModelName, InputFile inputFile) {
+ return FILE_ACCESS_FACTORIES.get(new Key(format, objectModelName)).readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. While data is written to
+ * the output file, this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param objectModelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <E> type of the engine-specific schema expected by the writer
+ * @return a configured writer builder for creating the appender
+ */
+ public static <E> WriteBuilder<?, E> writeBuilder(
+ FileFormat format, String objectModelName, EncryptedOutputFile outputFile) {
+ return ((FileAccessFactory<E>) FILE_ACCESS_FACTORIES.get(new Key(format, objectModelName)))
+ .writeBuilder(outputFile.encryptingOutputFile(), FileContent.DATA);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, these writers
+ * collect file metadata during the writing process and generate a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param objectModelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <E> type of the engine-specific schema expected by the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <E> DataWriteBuilder<?, E> dataWriteBuilder(
+ FileFormat format, String objectModelName, EncryptedOutputFile outputFile) {
+ return writeBuilderFor(format, objectModelName, outputFile, FileContent.DATA);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. These specialized writers collect equality
+ * delete records that identify rows to be deleted based on equality conditions, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param objectModelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <E> type of the engine-specific schema expected by the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <E> EqualityDeleteWriteBuilder<?, E> equalityDeleteWriteBuilder(
+ FileFormat format, String objectModelName, EncryptedOutputFile outputFile) {
+ return writeBuilderFor(format, objectModelName, outputFile, FileContent.EQUALITY_DELETES);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. These specialized writers collect {@link
+ * PositionDelete} records that identify rows to be deleted by file path and position, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param objectModelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <E> type of the engine-specific schema expected by the writer
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static <E> PositionDeleteWriteBuilder<?, E> positionDeleteWriteBuilder(
+ FileFormat format, String objectModelName, EncryptedOutputFile outputFile) {
+ return writeBuilderFor(format, objectModelName, outputFile, FileContent.POSITION_DELETES);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <B extends WriteBuilder<B, E>, E>
+ ContentFileWriteBuilderImpl<?, ?, E> writeBuilderFor(
+ FileFormat format,
+ String objectModelName,
+ EncryptedOutputFile outputFile,
+ FileContent content) {
+ return new ContentFileWriteBuilderImpl<>(
+ ((FileAccessFactory<E>) FILE_ACCESS_FACTORIES.get(new Key(format, objectModelName)))
+ .<B>writeBuilder(outputFile.encryptingOutputFile(), content),
+ outputFile.encryptingOutputFile().location(),
+ format);
+ }
+
+ /** Key used to identify readers and writers in the {@link FileAccessFactoryRegistry}. */
Review Comment:
Moved to using Pair.
Originally I have chickened out using the Pair because of the schema cache, but after double checking, I have realized that it is not used in our case.
Thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2100278943
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If the reader does not support encryption,
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (aad) prefix used for writing the file. If the reader
+ * does not support encryption, then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
Review Comment:
moved them to `fileAADPrefix` in both the `WriteBuilder` and the `ContentFileWriteBuilder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2099787997
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
Review Comment:
There are 2 main things:
- WriterContext - the writer needs the info if it is a data or a delete file we are writing. Based on this different table properties are used, like `write.delete.parquet.row-group-size-bytes`, or `write.parquet.row-group-size-bytes`
- The logic of writing the positional deletes are dependent on the file format `PositionDeleteStructWriter` for Parquet, `PositionAndRowDatumWriter` for Avro, `GenericOrcWriters.positionDelete` for ORC. They need different handling for different file formats, which I wanted to keep internal to the file formats themselves. But then they need the info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2175427527
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
Put together some tests, collected my thoughts and created a documentation about the way forward with the possible intermediate transformation layer. See: https://docs.google.com/document/d/1GdA8IowKMtS3QVdm8s-0X-ZRYetcHv2bhQ9mrSd3fd4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2098824679
##########
data/src/main/java/org/apache/iceberg/data/FileAccessFactoryRegistry.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAccessFactory;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified file access
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>File access factories are registered through {@link
+ * #registerFileAccessFactory(FileAccessFactory)} and used for creating readers and writers. Read
+ * builders are returned directly from the factory, while write builders may be wrapped in
+ * specialized content file writer implementations depending on the requested operation type.
+ */
+public final class FileAccessFactoryRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FileAccessFactoryRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, FileAccessFactory<?>> FILE_ACCESS_FACTORIES =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers a file access factory with this registry.
+ *
+ * <p>File access factories create readers and writers for specific combinations of file formats
+ * (Parquet, ORC, Avro) and object models ("generic", "spark", "flink", etc.). Registering custom
+ * factories allows integration of new data processing engines for the supported file formats with
+ * Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param fileAccessFactory the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FileAccessFactory#format()} and {@link FileAccessFactory#objectModeName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerFileAccessFactory(FileAccessFactory<?> fileAccessFactory) {
+ Key key = new Key(fileAccessFactory.format(), fileAccessFactory.objectModeName());
+ if (FILE_ACCESS_FACTORIES.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model %s clashes with %s. Both serves %s",
+ fileAccessFactory.getClass(), FILE_ACCESS_FACTORIES.get(key), key));
+ }
+
+ FILE_ACCESS_FACTORIES.put(key, fileAccessFactory);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FileAccessFactoryRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to engine-specific objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param objectModelName identifier for the engine-specific data representation (generic, spark,
+ * flink, etc.)
+ * @param inputFile source file to read data from
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static ReadBuilder<?> readBuilder(
+ FileFormat format, String objectModelName, InputFile inputFile) {
+ return FILE_ACCESS_FACTORIES.get(new Key(format, objectModelName)).readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. While data is written to
+ * the output file, this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param objectModelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <E> type of the engine-specific schema expected by the writer
+ * @return a configured writer builder for creating the appender
+ */
+ public static <E> WriteBuilder<?, E> writeBuilder(
+ FileFormat format, String objectModelName, EncryptedOutputFile outputFile) {
+ return ((FileAccessFactory<E>) FILE_ACCESS_FACTORIES.get(new Key(format, objectModelName)))
+ .writeBuilder(outputFile.encryptingOutputFile(), FileContent.DATA);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, these writers
+ * collect file metadata during the writing process and generate a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param objectModelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <E> type of the engine-specific schema expected by the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <E> DataWriteBuilder<?, E> dataWriteBuilder(
+ FileFormat format, String objectModelName, EncryptedOutputFile outputFile) {
+ return writeBuilderFor(format, objectModelName, outputFile, FileContent.DATA);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. These specialized writers collect equality
+ * delete records that identify rows to be deleted based on equality conditions, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param objectModelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <E> type of the engine-specific schema expected by the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <E> EqualityDeleteWriteBuilder<?, E> equalityDeleteWriteBuilder(
+ FileFormat format, String objectModelName, EncryptedOutputFile outputFile) {
+ return writeBuilderFor(format, objectModelName, outputFile, FileContent.EQUALITY_DELETES);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. These specialized writers collect {@link
+ * PositionDelete} records that identify rows to be deleted by file path and position, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param objectModelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <E> type of the engine-specific schema expected by the writer
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static <E> PositionDeleteWriteBuilder<?, E> positionDeleteWriteBuilder(
+ FileFormat format, String objectModelName, EncryptedOutputFile outputFile) {
+ return writeBuilderFor(format, objectModelName, outputFile, FileContent.POSITION_DELETES);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <B extends WriteBuilder<B, E>, E>
+ ContentFileWriteBuilderImpl<?, ?, E> writeBuilderFor(
+ FileFormat format,
+ String objectModelName,
+ EncryptedOutputFile outputFile,
+ FileContent content) {
+ return new ContentFileWriteBuilderImpl<>(
+ ((FileAccessFactory<E>) FILE_ACCESS_FACTORIES.get(new Key(format, objectModelName)))
+ .<B>writeBuilder(outputFile.encryptingOutputFile(), content),
+ outputFile.encryptingOutputFile().location(),
+ format);
+ }
+
+ /** Key used to identify readers and writers in the {@link FileAccessFactoryRegistry}. */
Review Comment:
Do we need a special class here? We already have Pair I think which basically does this and isn't private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2099755461
##########
data/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.DeleteSchemaUtil;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <C> the concrete builder type for method chaining
+ * @param <W> the type of the wrapped format-specific writer builder
+ * @param <E> the engine-specific schema type required by the writer
+ */
+@SuppressWarnings("unchecked")
+class ContentFileWriteBuilderImpl<
+ C extends ContentFileWriteBuilderImpl<C, W, E>, W extends WriteBuilder<W, E>, E>
+ implements DataWriteBuilder<C, E>,
+ EqualityDeleteWriteBuilder<C, E>,
+ PositionDeleteWriteBuilder<C, E> {
+ private final org.apache.iceberg.io.WriteBuilder<W, E> writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+ private Schema rowSchema = null;
+ private int[] equalityFieldIds = null;
+
+ ContentFileWriteBuilderImpl(
+ org.apache.iceberg.io.WriteBuilder<W, E> writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public C schema(Schema newSchema) {
+ writeBuilder.schema(newSchema);
+ return (C) this;
+ }
+
+ @Override
+ public C dataSchema(E engineSchema) {
+ writeBuilder.dataSchema(engineSchema);
+ return (C) this;
+ }
+
+ @Override
+ public C set(String property, String value) {
+ writeBuilder.set(property, value);
+ return (C) this;
+ }
+
+ @Override
+ public C set(Map<String, String> properties) {
+ properties.forEach(writeBuilder::set);
+ return (C) this;
+ }
+
+ @Override
+ public C meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return (C) this;
+ }
+
+ @Override
+ public C meta(Map<String, String> properties) {
+ properties.forEach(writeBuilder::meta);
+ return (C) this;
+ }
+
+ @Override
+ public C metricsConfig(MetricsConfig newMetricsConfig) {
+ writeBuilder.metricsConfig(newMetricsConfig);
+ return (C) this;
+ }
+
+ @Override
+ public C overwrite() {
+ writeBuilder.overwrite();
+ return (C) this;
+ }
+
+ @Override
+ public C fileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.fileEncryptionKey(encryptionKey);
+ return (C) this;
+ }
+
+ @Override
+ public C aadPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.aadPrefix(aadPrefix);
+ return (C) this;
+ }
+
+ @Override
+ public C rowSchema(Schema newSchema) {
+ this.rowSchema = newSchema;
+ return (C) this;
+ }
+
+ @Override
+ public C equalityFieldIds(List<Integer> fieldIds) {
+ this.equalityFieldIds = ArrayUtil.toIntArray(fieldIds);
+ return (C) this;
+ }
+
+ @Override
+ public C equalityFieldIds(int... fieldIds) {
+ this.equalityFieldIds = fieldIds;
+ return (C) this;
+ }
+
+ @Override
+ public C spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return (C) this;
+ }
+
+ @Override
+ public C partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return (C) this;
+ }
+
+ @Override
+ public C keyMetadata(EncryptionKeyMetadata metadata) {
+ this.keyMetadata = metadata;
+ return (C) this;
+ }
+
+ @Override
+ public C sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return (C) this;
+ }
+
+ @Override
+ public <D> DataWriter<D> dataWriter() throws IOException {
+ Preconditions.checkArgument(spec != null, "Cannot create data writer without spec");
+ Preconditions.checkArgument(
+ spec.isUnpartitioned() || partition != null,
+ "Partition must not be null when creating data writer for partitioned spec");
+
+ return new DataWriter<>(
+ writeBuilder.build(), format, location, spec, partition, keyMetadata, sortOrder);
+ }
+
+ @Override
+ public <D> EqualityDeleteWriter<D> equalityDeleteWriter() throws IOException {
+ Preconditions.checkState(
+ rowSchema != null, "Cannot create equality delete file without a schema");
+ Preconditions.checkState(
+ equalityFieldIds != null, "Cannot create equality delete file without delete field ids");
+ Preconditions.checkArgument(
+ spec != null, "Spec must not be null when creating equality delete writer");
+ Preconditions.checkArgument(
+ spec.isUnpartitioned() || partition != null,
+ "Partition must not be null for partitioned writes");
+
+ return new EqualityDeleteWriter<>(
+ writeBuilder
+ .schema(rowSchema)
+ .meta("delete-type", "equality")
Review Comment:
This will be the only place for these constants after the refactor and deprecation.
So I would keep them as it is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2066022181
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
+
+ /**
+ * Registers a new object model.
+ *
+ * @param objectModel the object model
+ * @throws IllegalArgumentException if an object model for the given {@code format} and {@code
+ * objectModelName} combination already exists
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerObjectModel(ObjectModel<?> objectModel) {
+ try {
+ Key key = new Key(objectModel.format(), objectModel.name());
+ if (OBJECT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model %s clashes with %s. Both serves %s",
+ objectModel.getClass(), OBJECT_MODELS.get(key), key));
+ }
+
+ OBJECT_MODELS.put(key, objectModel);
+ } catch (RuntimeException e) {
+ // failing to register an object model is normal and does not require a stack trace
+ LOG.info(
+ "Unable to use register object model {} for data files: {}", objectModel, e.getMessage());
Review Comment:
Fixed. Thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2066040201
##########
data/src/main/java/org/apache/iceberg/data/AppenderBuilder.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder for generating a {@link FileAppender}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E>
Review Comment:
Flink depends on `BaseTaskWriter` to implement the writers, and `BaseTaskWriter` needs `FileAppenderFactory` and `Appender`a to work.
Also there might be some other engines still depending on the `FileAppenderFactory` and `FileAppender` API which is not deprecated.
All-in-all, I think we need this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228065829
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param start the start position for this read
+ * @param length the length of the range this read should scan
+ */
+ B split(long start, long length);
+
+ /** Set the projection schema. */
+ B project(Schema schema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param saseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean saseSensitive);
Review Comment:
```suggestion
B caseSensitive(boolean caseSensitive);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228095998
##########
core/src/main/java/org/apache/iceberg/data/PositionDeleteWriteBuilder.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * A specialized builder for creating position-based delete file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link PositionDeleteWriter} instances.
+ *
+ * <p>The builder provides methods to configure the schema for the row data that might be included
+ * with the position deletes through {@link #rowSchema(Schema)}, enabling optional preservation of
+ * deleted record content.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the type of data records the writer will accept
+ */
+public interface PositionDeleteWriteBuilder<B extends PositionDeleteWriteBuilder<B, D>, D>
+ extends ContentFileWriteBuilder<B> {
+ /**
+ * Configures the schema for deleted row data to be stored in position delete files.
+ *
+ * <p>Position delete files can optionally store the content of deleted rows. When this schema is
+ * set, the writer will expect position delete records to include the complete row data of the
+ * records being deleted.
+ *
+ * <p>The row values provided to the writer should exactly match the structure specified via
+ * {@link #rowSchema(Schema)}.
+ *
+ * <p>If not configured, position delete files will contain only the path and position information
+ * without preserving the deleted record content.
+ *
+ * @param newRowSchema the Iceberg schema defining the structure for deleted row content
+ * @return this builder for method chaining
+ */
+ B rowSchema(Schema newRowSchema);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257284359
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
+
+ /** Set the file schema. */
+ B fileSchema(Schema fileSchema);
Review Comment:
Did this to differentiate from the `rowSchema`, but I was hesitating between the 2 anyways, so I have changed it to `schema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257265054
##########
core/src/main/java/org/apache/iceberg/data/PositionDeleteWriteBuilder.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * A specialized builder for creating position-based delete file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link PositionDeleteWriter} instances.
+ *
+ * <p>The builder provides methods to configure the schema for the row data that might be included
+ * with the position deletes through {@link #rowSchema(Schema)}, enabling optional preservation of
+ * deleted record content.
Review Comment:
The `FileWriterFactory` and the `FileAppenderFactory` implementations are using this method. If we remove the method, then we need to change that part of the API. I don't think we would like to do it without proper deprecation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254930405
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
Review Comment:
Looks like this isn't needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162702140
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
Review Comment:
This should not be ignored by default. Implementations should have to implement this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162712202
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
+ * @param <D> output type used for reading data, and input type for writing data and deletes
Review Comment:
What do we typically use for this type? Is it mostly D or mostly T?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2463937195
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Return the row type class for the object model implementation processed by this factory.
+ *
+ * <p>The model types act as a contract specifying the expected data structures for both reading
+ * (converting file formats into output objects) and writing (converting input objects into file
+ * formats). This ensures proper integration between Iceberg's storage layer and processing
+ * engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return the type of the data structures handled by this model implementation
+ */
+ Class<D> type();
+
+ Class<S> schemaType();
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2502921222
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.InputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Return the row type class for the object model implementation processed by this factory.
+ *
+ * <p>The model types act as a contract specifying the expected data structures for both reading
+ * (converting file formats into output objects) and writing (converting input objects into file
+ * formats). This ensures proper integration between Iceberg's storage layer and processing
+ * engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return the type of the data structures handled by this model implementation
+ */
+ Class<D> type();
+
+ /**
+ * Return the schema type class for the object model implementation processed by this factory.
+ *
+ * @return the type of the schema for the data structures handled by this model implementation
+ */
+ Class<S> schemaType();
+
+ /**
+ * Creates a writer builder for data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder
+ */
+ WriteBuilder<D, S> writeBuilder(EncryptedOutputFile outputFile);
Review Comment:
I followed exactly what @stevenzwu suggested.
Unfortunately, there are some hidden requirements in the Parquet implementation for encryption to work properly.
Parquet encryption was only functioning when the target was a `HadoopOutputFile`. Using `StandardEncryptedOutputFile.encryptingOutputFile()` produced an `AesGcmOutputFile`, which resulted in corrupt files.
Since Parquet only needed the Hadoop Configuration object from `HadoopOutputFile`, I addressed this by introducing:
- **HasConfiguration** interface ā to expose the Hadoop configuration (similar to `HadoopConfigurable`).
- **HadoopAesGcmOutputFile** ā which extends `AesGcmOutputFile`, but also stores the Hadoop configuration.
With these new classes, I made the following changes:
- Implemented `HasConfiguration` in both `HadoopAesGcmOutputFile` and `HadoopOutputFile`.
- Updated the check in the `Parquet.WriteBuilder` constructor to look for `HasConfiguration` instead of `HadoopOutputFile` when extracting the configuration.
- Modified `ParquetIO.file` to check for `HasConfiguration` instead of `HadoopOutputFile` when creating the `parquet.HadoopOutputFile`. - This one seems a bit risky for me, I would like to hear your thoughts
You can review the changes [here](https://github.com/apache/iceberg/pull/12298/commits/f1914a02b9cdc703ba8243cdaddca6cf660f3b62)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2503621087
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.InputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Return the row type class for the object model implementation processed by this factory.
+ *
+ * <p>The model types act as a contract specifying the expected data structures for both reading
+ * (converting file formats into output objects) and writing (converting input objects into file
+ * formats). This ensures proper integration between Iceberg's storage layer and processing
+ * engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return the type of the data structures handled by this model implementation
+ */
+ Class<D> type();
+
+ /**
+ * Return the schema type class for the object model implementation processed by this factory.
+ *
+ * @return the type of the schema for the data structures handled by this model implementation
+ */
+ Class<S> schemaType();
+
+ /**
+ * Creates a writer builder for data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder
+ */
+ WriteBuilder<D, S> writeBuilder(EncryptedOutputFile outputFile);
Review Comment:
Created a PR with a fix: https://github.com/apache/iceberg/pull/14528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2413218866
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+ Preconditions.checkArgument(
+ !FORMAT_MODELS.containsKey(key),
+ "Cannot register %s: %s is registered for format=%s model=%s",
+ formatModel.getClass(),
+ FORMAT_MODELS.containsKey(key) ? FORMAT_MODELS.get(key).getClass() : null,
+ key.first(),
+ key.second());
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D, S> WriteBuilder<D, S> writeBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, Object> factory = factoryForPositionDelete(format);
+ WriteBuilder<PositionDelete<?>, Object> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
+ return ((FormatModel<D, S>) FORMAT_MODELS.get(Pair.of(format, type)));
+ }
+
+ @SuppressWarnings("unchecked")
+ private static FormatModel<PositionDelete<?>, Object> factoryForPositionDelete(
+ FileFormat format) {
+ // Registry should use Pair<FileFormat, Class<?>> as key, so use PositionDelete.class
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#issuecomment-3862848753
I think this is ready and from our regular syncs as well as the discussion on the dev list, I don't think that there are any remaining blockers so I'll go ahead and merge it. That will unblock the file format and engine integrations, which can all happen in parallel after this core work is done.
Thanks @pvary!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2440877379
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
+ throw new UnsupportedOperationException("Not implemented yet");
+ }
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ ReadBuilder caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param filter the filter to set
+ */
+ ReadBuilder filter(Expression filter);
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ ReadBuilder set(String key, String value);
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default ReadBuilder setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /** Enables reusing the containers returned by the reader. Decreases pressure on GC. */
+ ReadBuilder reuseContainers();
+
+ /** Sets the batch size for vectorized readers. */
+ ReadBuilder recordsPerBatch(int numRowsPerBatch);
Review Comment:
Ok, that's reasonable, if the main driver is just to make it easy to move to the new APIs that's fair.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2445664919
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
Review Comment:
I am favoring removing it for now, and add it back when implementing shredding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2447610349
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
Review Comment:
In previous versions, the code prevented re-registering `FormatModel` instances with the same key. We removed this restriction to accommodate scenarios where engines like Flink and Spark may reload classes using different classloaders during job restarts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2447617865
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
Review Comment:
Removed the method for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2447615999
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2508148899
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.InputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Return the row type class for the object model implementation processed by this factory.
+ *
+ * <p>The model types act as a contract specifying the expected data structures for both reading
+ * (converting file formats into output objects) and writing (converting input objects into file
+ * formats). This ensures proper integration between Iceberg's storage layer and processing
+ * engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return the type of the data structures handled by this model implementation
+ */
+ Class<D> type();
+
+ /**
+ * Return the schema type class for the object model implementation processed by this factory.
+ *
+ * @return the type of the schema for the data structures handled by this model implementation
+ */
+ Class<S> schemaType();
+
+ /**
+ * Creates a writer builder for data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder
+ */
+ WriteBuilder<D, S> writeBuilder(EncryptedOutputFile outputFile);
Review Comment:
@ggershinsky: Could you please help me with unit tests which execute the different encryption/decryption methods, so I can check what is happening?
I suspect that the Parquet.write expects and handles `NativeEncryptingOutputFile` for writes, as there explicit `implements` checks in the `Parquet.java`
`StandardEncryptedOutputFile` also implements `NativeEncryptingOutputFile`. I guess this is why it is handled correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2057631093
##########
data/src/main/java/org/apache/iceberg/data/AppenderBuilder.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder for generating a {@link FileAppender}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E>
Review Comment:
IIRC, `Appender` does not produce Iceberg metadata like `DataFile` and `DeleteFile`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095463240
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If encryption is not supported by the reader
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data prefix used for writing the file. If encryption is not
Review Comment:
done
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If encryption is not supported by the reader
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data prefix used for writing the file. If encryption is not
+ * supported by the reader then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the engine native schema for the input. Defines the input type when there is N to 1
+ * mapping between the engine type and the Iceberg type, and providing the Iceberg schema is not
+ * enough for the conversion.
+ */
+ B engineSchema(E newEngineSchema);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2100237439
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If the reader does not support encryption,
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (aad) prefix used for writing the file. If the reader
+ * does not support encryption, then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the engine-specific schema for the input data records.
+ *
+ * <p>This method is necessary when the mapping between engine types and Iceberg types is not
+ * one-to-one. For example, when multiple engine types could map to the same Iceberg type, or when
+ * schema metadata beyond the structure is needed to properly interpret the data.
+ *
+ * <p>While the Iceberg schema defines the expected output structure, the engine schema provides
+ * the exact input format details needed for proper type conversion.
+ *
+ * @param newEngineSchema the native schema representation from the engine (Spark, Flink, etc.)
+ * @return this builder for method chaining
+ */
+ B dataSchema(E newEngineSchema);
+
+ /** Finalizes the configuration and builds the {@link FileAppender}. */
+ <D> FileAppender<D> build() throws IOException;
Review Comment:
The `WriteBuilder` appender will change the input type based on the ContentType parameter passed on the constructor. If `D` is the type of the object model records, then for `DATA` and `EQUALITY_DELETES` the appender will expect `D`, but for `POSITION_DELETES` it will expect `PositionDelete<D>`.
Also this would change the method signature for the existing ORC/Parquet/Avro`<D> CloseableIterable<D> ReadBuilder.build()`, <D> FileAppender<D> WriteBuilder.build()` methods. This makes creating backward compatible change even more convoluted, as we will need to create a new class, and map all the calls in the old one.
We could explore this direction as well, but this deviates even further from the current solution, and we need strong community approval for it. One possibility what I can see is to extend the `FileAccessFactory` API:
```
public interface FileAccessFactory<E, D> {
FileFormat format();
String objectModeName();
<B extends ReadBuilder<B, D>> B readBuilder(InputFile inputFile);
// For Writing D (Data and equality delete)
<B extends WriteBuilder<B, E, D>> B writeBuilder(OutputFile outputFile, FileContent content);
// For Writing PositionDelete<D>
<B extends WriteBuilder<B, E, PositionDelete<D>>> B positionDeleteWriteBuilder(OutputFile outputFile, FileContent content);
}
```
I agree that the API would be nicer/cleaner. My question is "does it worth it"? Will I have reviewers how check the changes for this level of change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254840553
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param start the start position for this read
+ * @param length the length of the range this read should scan
+ */
+ B split(long start, long length);
+
+ /** Set the projection schema. */
+ B project(Schema schema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param filter the filter to set
+ */
+ B filter(Expression filter);
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ B set(String key, String value);
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ @SuppressWarnings("unchecked")
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /** Enables reusing the containers returned by the reader. Decreases pressure on GC. */
+ B reuseContainers();
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata and not coming from the data files themselves. The keys of the map are the column
+ * ids, the values are the accessors generating the values.
Review Comment:
If the values are accessors, then the value type should be the accessor, not `?`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254853428
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param start the start position for this read
+ * @param length the length of the range this read should scan
+ */
+ B split(long start, long length);
+
+ /** Set the projection schema. */
+ B project(Schema schema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param filter the filter to set
+ */
+ B filter(Expression filter);
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ B set(String key, String value);
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ @SuppressWarnings("unchecked")
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
Review Comment:
This API should not use unchecked casts like this. Instead, add a `self()` method that returns the builder type and is called in places like this. You can see how this works in [`SnapshotProducer`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotProducer.java#L148)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254989845
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
Review Comment:
I think there is probably a better name than `ContentFileWriteBuilder`. The term `ContentFile` is just a generic internal name that we use as a base for both `DataFile` and `DeleteFile` and it isn't a well-defined concept in the format. Leaking it into exposed APIs like this is not a good use. We are also going to be refactoring those classes for v4 metadata.
This interface is also a little strange. it looks like it has all of the methods from `WriteBuilder` except for `build`. It adds 4 methods to set metadata that get passed into `DataFile` and `DeleteFile` metadata: `spec`, `partition`, `key_metadata`, and `sortOrder`. I wonder if this could just be an interface with those last 4 that gets added to `DataWriteBuilder` and the delete builders instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257233974
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param start the start position for this read
+ * @param length the length of the range this read should scan
+ */
+ B split(long start, long length);
+
+ /** Set the projection schema. */
+ B project(Schema schema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param filter the filter to set
+ */
+ B filter(Expression filter);
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ B set(String key, String value);
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ @SuppressWarnings("unchecked")
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257569287
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.dataBuilder(outputFile.encryptingOutputFile());
Review Comment:
The `dataBuilder` returns a `WriteBuilder`, which doesn't have the `partition`, `spec` etc. methods. Those methods are only available in the `ContentFileWriteBuilder` API.
So we don't expose additional methods here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228049976
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ B project(Schema newSchema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param newCaseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean newCaseSensitive);
Review Comment:
moved to `caseSensitive`
The classes overriding this have to change the parameter name (as typically the builder attribute name is `caseSensitive`), and reusing the attribute name in a method parameter name is forbidden by checkstyle.
But agree, that the API should have the nicer names š
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228057176
##########
core/src/main/java/org/apache/iceberg/io/FormatModel.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ */
+public interface FormatModel<D> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
+ *
+ * <p>The model names (for example: "generic", "spark", "spark-vectorized", "flink", "arrow") act
+ * as a contract specifying the expected data structures for both reading (converting file formats
+ * into output objects) and writing (converting input objects into file formats). This ensures
+ * proper integration between Iceberg's storage layer and processing engines.
+ *
+ * @return string identifier for this model implementation
+ */
+ String modelName();
Review Comment:
As discussed offline, updated the comment to make it clear that the `modelName` is string, so engines could register their own models.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228067056
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
+
+ /** Set the file schema. */
+ B fileSchema(Schema newSchema);
Review Comment:
```suggestion
B fileSchema(Schema schema);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2772890461
##########
core/src/main/java/org/apache/iceberg/deletes/PositionDeleteWriter.java:
##########
@@ -60,7 +60,7 @@ public class PositionDeleteWriter<T> implements FileWriter<PositionDelete<T>, De
private DeleteFile deleteFile = null;
public PositionDeleteWriter(
- FileAppender<StructLike> appender,
+ FileAppender<PositionDelete<T>> appender,
Review Comment:
Done.
As suggested, just deprecated the old constructor and noted that the appender will change.
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and {@link
+ * FileWriterBuilder}s for writing various types of Iceberg content files. The appropriate builder
+ * is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private FormatModelRegistry() {}
+
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static synchronized void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null,
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @param <S> the type of the output schema for the reader
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<? extends D> type, InputFile inputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return model.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> FileWriterBuilder<DataWriter<D>, S> dataWriteBuilder(
+ FileFormat format, Class<? extends D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return FileWriterBuilderImpl.forDataFile(model, outputFile);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> FileWriterBuilder<EqualityDeleteWriter<D>, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return FileWriterBuilderImpl.forEqualityDelete(model, outputFile);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public static <D> FileWriterBuilder<PositionDeleteWriter<D>, ?> positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<D>, ?> model =
+ (FormatModel<PositionDelete<D>, ?>) (FormatModel) modelFor(format, PositionDelete.class);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775988319
##########
core/src/main/java/org/apache/iceberg/formats/FileWriterBuilder.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriter;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <W> the concrete writer type the builder produces
+ * @param <S> the type of the schema for the input data
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/formats/FileWriterBuilder.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriter;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <W> the concrete writer type the builder produces
+ * @param <S> the type of the schema for the input data
+ */
+public interface FileWriterBuilder<W extends FileWriter<?, ?>, S> {
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> meta(String property, String value);
+
+ /**
+ * Add the new properties to file metadata for the created file.
+ *
+ * @param properties a map of file metadata properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ FileWriterBuilder<W, S> metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ FileWriterBuilder<W, S> overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the writer
+ * does not support encryption, then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withAADPrefix(ByteBuffer aadPrefix);
+
+ /** Sets the partition specification for the Iceberg metadata. */
+ FileWriterBuilder<W, S> spec(PartitionSpec newSpec);
+
+ /** Sets the partition value for the Iceberg metadata. */
+ FileWriterBuilder<W, S> partition(StructLike partition);
+
+ /** Sets the encryption key metadata for Iceberg metadata. */
+ FileWriterBuilder<W, S> keyMetadata(EncryptionKeyMetadata keyMetadata);
+
+ /** Sets the sort order for the Iceberg metadata. */
+ FileWriterBuilder<W, S> sortOrder(SortOrder sortOrder);
+
+ /** Set the file schema. */
+ FileWriterBuilder<W, S> schema(Schema schema);
+
+ /**
+ * Sets the engine's representation accepted by the writer.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775990547
##########
core/src/main/java/org/apache/iceberg/formats/FileWriterBuilder.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriter;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <W> the concrete writer type the builder produces
+ * @param <S> the type of the schema for the input data
+ */
+public interface FileWriterBuilder<W extends FileWriter<?, ?>, S> {
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> meta(String property, String value);
+
+ /**
+ * Add the new properties to file metadata for the created file.
+ *
+ * @param properties a map of file metadata properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ FileWriterBuilder<W, S> metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ FileWriterBuilder<W, S> overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the writer
+ * does not support encryption, then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withAADPrefix(ByteBuffer aadPrefix);
+
+ /** Sets the partition specification for the Iceberg metadata. */
+ FileWriterBuilder<W, S> spec(PartitionSpec newSpec);
+
+ /** Sets the partition value for the Iceberg metadata. */
+ FileWriterBuilder<W, S> partition(StructLike partition);
+
+ /** Sets the encryption key metadata for Iceberg metadata. */
+ FileWriterBuilder<W, S> keyMetadata(EncryptionKeyMetadata keyMetadata);
+
+ /** Sets the sort order for the Iceberg metadata. */
+ FileWriterBuilder<W, S> sortOrder(SortOrder sortOrder);
+
+ /** Set the file schema. */
+ FileWriterBuilder<W, S> schema(Schema schema);
+
+ /**
+ * Sets the engine's representation accepted by the writer.
+ *
+ * <p>Some data types require additional type information from the engine schema that cannot be
+ * fully expressed by the Iceberg schema or the data itself. For example, a variant type may use a
+ * shredded representation that relies on engine-specific metadata to map back to the Iceberg
+ * schema.
+ *
+ * <p>The engine schema must be aligned with the Iceberg schema, but may include representation
+ * details that Iceberg considers equivalent.
Review Comment:
Is it enough here, or shall we add it somewhere else too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162683580
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ */
+public interface FileAccessFactory<E, D> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
+ *
+ * <p>The object model names (such as "generic", "spark", "spark-vectorized", "flink", "arrow")
Review Comment:
As above, I don't think that specific object models should be listed here. Also, if they are listed anywhere they should be listed once rather than in repeated lists that can get out of sync with the code and one another.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163604826
##########
build.gradle:
##########
@@ -373,6 +373,7 @@ project(':iceberg-data') {
dependencies {
implementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
api project(':iceberg-api')
+ implementation project(':iceberg-common')
Review Comment:
DynMethods in the `ObjectModelRegistry`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162569005
##########
api/src/main/java/org/apache/iceberg/io/FileReader.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.util.Map;
+
+/**
+ * A reader capable of reading files and returning records of type {@code D}.
+ *
+ * <p>This interface should be implemented by classes that not only read records from files but also
+ * provide metadata stored in the file.
+ *
+ * @param <D> the record type
+ */
+public interface FileReader<D> extends CloseableIterable<D> {
+ Map<String, String> meta();
Review Comment:
I don't think that we should add this and should instead track down and remove cases where we are using the file metadata to recover a partition spec. I think that only happens for unused and deprecated calls in public methods exposed in the core module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163711276
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link ReadBuilder} instances based on:
+ *
+ * <ul>
+ * <li>source file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * </ul>
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+public interface ReadBuilder<B extends ReadBuilder<B>> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
Review Comment:
reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2726838278
##########
core/src/main/java/org/apache/iceberg/formats/CommonWriteBuilder.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface CommonWriteBuilder<B extends CommonWriteBuilder<B>> {
Review Comment:
What API stability do we guarantee for the new read/write interfaces?
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
Review Comment:
I would prefer if we returned the MODELS map here instead of using a void return type with side effects. This makes testing easier.
```java
MODELS = loadSupportedFormats();
```
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static synchronized void register(FormatModel<?, ?> formatModel) {
Review Comment:
Instead of synchronizing the whole method, could we register access to the critical path:
```java
synchronized (FormatModelRegister.class) {
MODELS.put(key, formatModel)
}
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static synchronized void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null,
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @param <S> the type of the output schema for the reader
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<? extends D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<? extends D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return CommonWriteBuilderImpl.forDataFile(
+ factory.writeBuilder(outputFile), outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return CommonWriteBuilderImpl.forEqualityDelete(
+ factory.writeBuilder(outputFile), outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ @SuppressWarnings("rawtypes")
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete, ?> factory = factoryFor(format, PositionDelete.class);
+ return CommonWriteBuilderImpl.forPositionDelete(
+ factory.writeBuilder(outputFile), outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @VisibleForTesting
+ static Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> models() {
+ return MODELS;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<? extends D> type) {
+ FormatModel<D, S> model = (FormatModel<D, S>) MODELS.get(Pair.of(format, type));
+ Preconditions.checkArgument(
+ model != null, "Format model is not registered for format %s and type %s", format, type);
+ return model;
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
Review Comment:
Could we return the map here? This will also make testing easier.
```java
private static Map<..> loadSupportedFormats() {
```
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
Review Comment:
Should there be an API to add new CLASSES_TO_REGISTER? For example, when you want an application register a custom reader/writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2276209372
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
Review Comment:
The `ContentFileWriteBuilder` is not exposed, by I'm open for any suggestions š
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2440846191
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
Review Comment:
This is something which was modified multiple times.
`(long start, long length)` could be enough, but when the implementation comes, and the `start` and `length` will be used, then checkstyle will complain for reusing/shading attributes with the same name:
```
public ReadBuilder split(long start, long length) {
this.start = start;
this.length = length;
return this;
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2337986921
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.DeleteSchemaUtil;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the type of data records the writer will accept
Review Comment:
Missing doc for `@param <S>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2413221369
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+ Preconditions.checkArgument(
+ !FORMAT_MODELS.containsKey(key),
+ "Cannot register %s: %s is registered for format=%s model=%s",
+ formatModel.getClass(),
+ FORMAT_MODELS.containsKey(key) ? FORMAT_MODELS.get(key).getClass() : null,
+ key.first(),
+ key.second());
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D, S> WriteBuilder<D, S> writeBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, Object> factory = factoryForPositionDelete(format);
+ WriteBuilder<PositionDelete<?>, Object> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
Review Comment:
Added a null check. Please take a look at the error message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2376949174
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+ Preconditions.checkArgument(
+ !FORMAT_MODELS.containsKey(key),
+ "Cannot register %s: %s is registered for format=%s model=%s",
+ formatModel.getClass(),
+ FORMAT_MODELS.containsKey(key) ? FORMAT_MODELS.get(key).getClass() : null,
+ key.first(),
+ key.second());
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D, S> WriteBuilder<D, S> writeBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, Object> factory = factoryForPositionDelete(format);
+ WriteBuilder<PositionDelete<?>, Object> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
+ return ((FormatModel<D, S>) FORMAT_MODELS.get(Pair.of(format, type)));
+ }
+
+ @SuppressWarnings("unchecked")
+ private static FormatModel<PositionDelete<?>, Object> factoryForPositionDelete(
+ FileFormat format) {
+ // Registry should use Pair<FileFormat, Class<?>> as key, so use PositionDelete.class
Review Comment:
nit: this comment seems unnecessary
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
Review Comment:
if we are going to check class name, we probably should use fully qualified `getCanonicalName`.
I guess we can't use the equality comparison if they are loaded from different class loaders?
```
model1.getClass() == model2.getClass()
```
Also if the fully qualified class names match, is it necessarily to check the fields?
##########
core/src/main/java/org/apache/iceberg/formats/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <S> the type of the schema for the input data
+ * @param <D> the type of data records the writer will accept
+ */
+@SuppressWarnings("unchecked")
+abstract class ContentFileWriteBuilderImpl<B extends ContentFileWriteBuilder<B, S>, D, S>
+ implements ContentFileWriteBuilder<B, S> {
+ private final WriteBuilder<D, S> writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ static <D, S> DataWriteBuilder<D, S> forDataFile(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new DataFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> EqualityDeleteWriteBuilder<D, S> forEqualityDelete(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new EqualityDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> PositionDeleteWriteBuilder<D, S> forPositionDelete(
+ WriteBuilder<PositionDelete<D>, S> writeBuilder, String location, FileFormat format) {
+ return new PositionDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ private ContentFileWriteBuilderImpl(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public B set(String property, String value) {
+ writeBuilder.set(property, value);
+ return self();
+ }
+
+ @Override
+ public B meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return self();
+ }
+
+ @Override
+ public B metricsConfig(MetricsConfig metricsConfig) {
+ writeBuilder.metricsConfig(metricsConfig);
+ return self();
+ }
+
+ @Override
+ public B overwrite() {
+ writeBuilder.overwrite();
+ return self();
+ }
+
+ @Override
+ public B fileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.fileEncryptionKey(encryptionKey);
+ return self();
+ }
+
+ @Override
+ public B fileAADPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.fileAADPrefix(aadPrefix);
+ return self();
+ }
+
+ @Override
+ public B spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return self();
+ }
+
+ @Override
+ public B partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return self();
+ }
+
+ @Override
+ public B keyMetadata(EncryptionKeyMetadata newKeyMetadata) {
+ this.keyMetadata = newKeyMetadata;
+ return self();
+ }
+
+ @Override
+ public B sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return self();
+ }
+
+ private static class DataFileWriteBuilder<D, S>
Review Comment:
close this thread since most think it is good to keep builders as interfaces.
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+ Preconditions.checkArgument(
+ !FORMAT_MODELS.containsKey(key),
+ "Cannot register %s: %s is registered for format=%s model=%s",
+ formatModel.getClass(),
+ FORMAT_MODELS.containsKey(key) ? FORMAT_MODELS.get(key).getClass() : null,
+ key.first(),
+ key.second());
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D, S> WriteBuilder<D, S> writeBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, Object> factory = factoryForPositionDelete(format);
+ WriteBuilder<PositionDelete<?>, Object> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
Review Comment:
The two `factoryFor` methods should throw illegal argument exception for not supported file format and row type.
In the uber PR, I can see the registry can fail silently for class not found exception. Maybe the error msg here could also explain some possible reasons, like the relevant classes (Orc) are not loaded.
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()} ()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+ Preconditions.checkArgument(
Review Comment:
closing old thread as code is updated to fail re-registering for a different format model class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2066004093
##########
data/src/main/java/org/apache/iceberg/data/PositionDeleteWriterBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * Builder for generating an {@link PositionDeleteWriter}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface PositionDeleteWriterBuilder<B extends PositionDeleteWriterBuilder<B, E>, E>
+ extends FileWriterBuilderBase<B, E> {
+ /** Sets the row schema for the delete writers. */
+ B withRowSchema(Schema newSchema);
+
Review Comment:
The deletes are always expected to be sorted. The writer/appender doesn't sort them. There are several abstraction layers above the actual writers which do the sorting, but at this level we always expect sorted input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2068307219
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
+
+ /**
+ * Registers a new object model.
+ *
+ * @param objectModel the object model
+ * @throws IllegalArgumentException if an object model for the given {@code format} and {@code
+ * objectModelName} combination already exists
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerObjectModel(ObjectModel<?> objectModel) {
Review Comment:
Ehh.. that was a mistake on my side.
The `try/catch` is not needed after the refactors.
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2073415112
##########
data/src/main/java/org/apache/iceberg/data/AppenderBuilder.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder for generating a {@link FileAppender}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E>
Review Comment:
Tried my hands on moving the WriterMode to the constructor for the appender, but that would mean even more serious refactor of the old codebase. I'm happy to do it, if the community thinks that is the direction moving forward, but currently I'm struggling to find reviewers for the current code. I would be reluctant to add any more big changes unless specifically asked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2077373497
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -36,10 +36,10 @@
/**
* Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
- * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
- * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
- * and the requested `object model name` the registry returns the correct reader and writer
- * builders. These builders could be used to generate the readers and writers.
+ * AppenderBuilder}, {@link DataWriterBuilder}, {@link EqualityDeleteWriterBuilder}, {@link
Review Comment:
Moved a bit.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095300900
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * File formats should implement this interface to provide a builder for reading data files. {@link
+ * ReadBuilder} reads the data files with the specified parameters. The returned objects are defined
+ * by the {@link ObjectModel} which is used to read the data.
+ *
+ * <p>This interface is directly exposed for the users to parameterize readers.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ */
+public interface ReadBuilder<B extends ReadBuilder<B>> {
+ /** The configuration key for the batch size in case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the expression. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user provided setting if is not supported by them.
+ */
+ default B reuseContainers() {
+ // Skip container reuse configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata, and not coming from the data files themselves. The keys of the map are the
+ * column ids, the values are the accessors generating the values.
+ */
+ B constantFieldAccessors(Map<Integer, ?> constantFieldAccessors);
+
+ /** Sets a mapping from external schema names to Iceberg type IDs. */
+ B withNameMapping(NameMapping newNameMapping);
+
+ /**
+ * Sets the file encryption key used for reading the file. If encryption is not supported by the
+ * reader then an exception should be thrown.
+ */
+ default B withFileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data prefix for encryption. If encryption is not supported
+ * by the reader then an exception should be thrown.
+ */
+ default B withAADPrefix(ByteBuffer aadPrefix) {
Review Comment:
These are the methods which are not pushed down to the `Appender`. They were different in the previous API, and I have kept it so, but I'm open to change it. This would mean more code change when migrating to the new API, but I am fine if the community decides so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2096231250
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
I understand the concern of including `Data` would that it might be confused with `Data/PositionDelete/EqualityDelete` writer.
maybe `FileAccessFactory`. access could cover both read and write/appender.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#issuecomment-2895768606
@huaxingao, @pvary Could you take a look from a comet prospective? I know you have some custom code that would be using this as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2099753128
##########
data/src/main/java/org/apache/iceberg/data/PositionDeleteWriteBuilder.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * A specialized builder for creating position-based delete file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link PositionDeleteWriter} instances.
+ *
+ * <p>The builder provides methods to configure the schema for the row data that might be included
+ * with the position deletes through {@link #rowSchema(Schema)}, enabling optional preservation of
+ * deleted record content.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type required by the writer for data conversion
+ */
+public interface PositionDeleteWriteBuilder<B extends PositionDeleteWriteBuilder<B, E>, E>
+ extends ContentFileWriteBuilder<B, E> {
+ /** Sets the row schema for the delete writers. */
+ B rowSchema(Schema newSchema);
Review Comment:
Discussed with @RussellSpitzer, and we can't deprecate deletes for now. We will support V2 tables for quite a while, so we need this on the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2100272067
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
Review Comment:
Actually the `WriteBuilder` has this responsibility. The file format specific code converts the Iceberg schema to the file format specific schema. In many cases (not in Avro) the file format specific schema is passed to the engine specific code and use as a base for conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2098728163
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If the reader does not support encryption,
Review Comment:
```suggestion
* Sets the encryption key used for writing the file. If the writer does not support encryption,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162695074
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ */
+public interface FileAccessFactory<E, D> {
Review Comment:
It's a best practice to avoid the name `Factory` unless the class is actually used as a factory, meaning that it is created with configuration and then used to create objects using that configuration similar to how the `FileWriterFactory` is used. In addition, it isn't clear what "access" means here and I would expect the class to have something to do with _accessing_ files rather than creating builders for a specific object model.
I see that there was [a thread](https://github.com/apache/iceberg/pull/12774#discussion_r2093626096) about using `ObjectModel` and this was renamed. I understand the criticism of `ObjectModel`, but I think that the rename adds to the confusion instead of making it more understandable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162696513
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ */
+public interface FileAccessFactory<E, D> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
+ *
+ * <p>The object model names (such as "generic", "spark", "spark-vectorized", "flink", "arrow")
+ * act as a contract specifying the expected data structures for both reading (converting file
+ * formats into output objects) and writing (converting input objects into file formats). This
+ * ensures proper integration between Iceberg's storage layer and processing engines.
+ *
+ * @return string identifier for this object model implementation
+ */
+ String objectModelName();
+
+ /**
+ * Creates a writer builder for standard data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory for regular data content.
+ *
+ * <p>The builder follows the fluent pattern for configuring writer properties like compression,
+ * encryption, row group size, and other format-specific options.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder for standard data files
+ * @param <B> the concrete builder type for method chaining
+ */
+ <B extends WriteBuilder<B, E, D>> B dataWriteBuilder(OutputFile outputFile);
Review Comment:
The existing builders are created using `writeData` and `writeDeletes`. Why are the names longer here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163715964
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ */
+public interface FileAccessFactory<E, D> {
Review Comment:
This is a sizable change. I will first revisit https://github.com/apache/iceberg/pull/12774#discussion_r2162731545, and will revert if we still need this
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
Review Comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163713034
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
Review Comment:
Removed the list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2164138195
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
> > Another possibility could be that we define an intermediate Object Model (maybe something like Arrow), and provide a double transformation File Format -> Arrow -> Engine, and Engine -> Arrow -> File Format.[..]
>
> This is worth exploring. It may seem like it would be slower, but vectorized reads are much faster so I think it would be better overall as long as we could adapt to the right object model. It may be slightly slower for Avro (that can't vectorize reads and can produce specific classes) but overall I think it would be a win. Plus we would be able to consolidate reader code and focus on a shared high performance vectorized path.
I was considering this as one of the next steps. The PR is already huge. If we start with the whole read/write path refactoring this will quickly becomes unmanageable. That is why I decided to move in smaller steps. I'm afraid of not getting reviews for an even bigger change.
If we agree that this one is a good goal, I'm happy to explore and commit to work on the next steps.
Regardless, I think the standardization of the reader and the writer api is a good first step.
> > I don't see how we can push this behind a meaningful common interface.
>
> The path to a better interface is to reduce the complexity of the object models that get built in the reader and writer functions. For Avro and Parquet (the ones I'm familiar with) there are different schemas passed to build the object model readers and writers for a couple of reasons. First, the format-specific schema is used so that the reader or writer is aligned exactly with the file schema (if Avro has an option, the option ID byte has to be written or read). Second, the engine-specific schema is used so that the correct engine type method (like `InternalRow#getByte`) is called. There is a possibility that we could add an adapter to remove engine-specific types (implement `InternalRow#getInt` that calls `InternalRow#getByte` instead of requiring the writer to do it). And lastly, the Iceberg schema is needed in some cases where we don't have information in the file-specific schema or when we need it to construct a generic record object. The former is why we recently added th
e Iceberg schema to Parquet readers -- Parquet doesn't have a Variant annotation yet.
>
> There may be a path forward where we solve these challenges, or where the API is not based on passing potentially all 3 schemas into the reader/writer build functions. @pvary have you explored simplifying this?
I have spent serious time on trying to simplify the current read path. One of the constraints was to keep the input of the writers and the output of the readers intact, so if some external users depend on our readers/writers they could change the parametrization, but keep the other part of the code without changes. Another constraint was to not change anything below the reader/writer functions.
If we are ready to change that part as well, then we need the following information as a "physical" constraint:
- Data File schema
- Input data schema
We can simplify the user input to:
- Requesting the Iceberg schema
- Converter from the Iceberg schema to the Data File schema
- Converter from the Iceberg schema to the Engine schema
- Requesting the user to exactly match the input to the calculated Engine schema
Again, this is something which is even bigger than the current PR. If we agree that this one is a good goal, I'm happy to explore and commit to work on this in the next PR. I need some time, and I need reviewers, but I would be more than happy to work on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2103603849
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If the reader does not support encryption,
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (aad) prefix used for writing the file. If the reader
+ * does not support encryption, then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the engine-specific schema for the input data records.
+ *
+ * <p>This method is necessary when the mapping between engine types and Iceberg types is not
+ * one-to-one. For example, when multiple engine types could map to the same Iceberg type, or when
+ * schema metadata beyond the structure is needed to properly interpret the data.
+ *
+ * <p>While the Iceberg schema defines the expected output structure, the engine schema provides
+ * the exact input format details needed for proper type conversion.
+ *
+ * @param newEngineSchema the native schema representation from the engine (Spark, Flink, etc.)
+ * @return this builder for method chaining
+ */
+ B dataSchema(E newEngineSchema);
+
+ /** Finalizes the configuration and builds the {@link FileAppender}. */
+ <D> FileAppender<D> build() throws IOException;
Review Comment:
the `FileAccessFactoryRegistry` already have different write builder methods for different content type (data, equality, position delete). it seems reasonable to have similar model for the `FileAccessFactory` too.
Anyway, I need to spend more time to better understand the current class hierarchy to further review the overall work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2227774387
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ B project(Schema newSchema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param newCaseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean newCaseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ B filter(Expression newFilter);
Review Comment:
would be good to also omit the `new` prefix here and in a few other methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254829659
##########
core/src/main/java/org/apache/iceberg/data/EqualityDeleteWriteBuilder.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+
+/**
+ * A specialized builder for creating equality-based delete file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link EqualityDeleteWriter} instances.
+ *
+ * <p>The builder provides methods to configure which fields should be used for equality comparison
+ * through {@link #equalityFieldIds(List)} or {@link #equalityFieldIds(int...)}, along with schema
+ * configuration for the delete records.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the type of data records the writer will accept
+ */
+public interface EqualityDeleteWriteBuilder<B extends EqualityDeleteWriteBuilder<B, D>, D>
+ extends ContentFileWriteBuilder<B> {
+ /** Sets the row schema for the delete writers. */
+ B rowSchema(Schema rowSchema);
+
+ /** Sets the equality field ids for the equality delete writer. */
+ B equalityFieldIds(List<Integer> fieldIds);
+
+ /** Sets the equality field ids for the equality delete writer. */
+ B equalityFieldIds(int... fieldIds);
Review Comment:
Convenience method overloads should have default implementations. There's no need for implementations to re-implement this only to call `equalityFieldIds(List)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257499755
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2258040683
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
Review Comment:
The `ContentFileWriteBuilder` is not exposed. It is a package private interface.
Only it's methods are exposed through the specific builders, like `DataWriteBuilder`, `EqualityDeleteWriteBuilder` and `PositionDeleteWriteBuilder`.
What I can do is copy all of the methods, and implementations to the specific builders, but I found the code duplications ugly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2372924011
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <S> the type of the schema for the input data
+ * @param <D> the type of data records the writer will accept
+ */
+@SuppressWarnings("unchecked")
+abstract class ContentFileWriteBuilderImpl<B extends ContentFileWriteBuilder<B, S>, D, S>
+ implements ContentFileWriteBuilder<B, S> {
+ private final WriteBuilder<D, S> writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ static <D, S> DataWriteBuilder<D, S> forDataFile(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new DataFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> EqualityDeleteWriteBuilder<D, S> forEqualityDelete(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new EqualityDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> PositionDeleteWriteBuilder<D, S> forPositionDelete(
+ WriteBuilder<PositionDelete<D>, S> writeBuilder, String location, FileFormat format) {
+ return new PositionDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ private ContentFileWriteBuilderImpl(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public B set(String property, String value) {
+ writeBuilder.set(property, value);
+ return self();
+ }
+
+ @Override
+ public B meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return self();
+ }
+
+ @Override
+ public B metricsConfig(MetricsConfig metricsConfig) {
+ writeBuilder.metricsConfig(metricsConfig);
+ return self();
+ }
+
+ @Override
+ public B overwrite() {
+ writeBuilder.overwrite();
+ return self();
+ }
+
+ @Override
+ public B fileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.fileEncryptionKey(encryptionKey);
+ return self();
+ }
+
+ @Override
+ public B fileAADPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.fileAADPrefix(aadPrefix);
+ return self();
+ }
+
+ @Override
+ public B spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return self();
+ }
+
+ @Override
+ public B partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return self();
+ }
+
+ @Override
+ public B keyMetadata(EncryptionKeyMetadata newKeyMetadata) {
+ this.keyMetadata = newKeyMetadata;
+ return self();
+ }
+
+ @Override
+ public B sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return self();
+ }
+
+ private static class DataFileWriteBuilder<D, S>
Review Comment:
> I would want to keep the DataWriteBuilder, EqualityDeleteWriteBuilder and PositionDeleteWriteBuilder.
Agree that these need to be public and exposed. But do they need to be interfaces? They can just be concrete classes. Usually, builders are not interfaces.
If they can be classes, they can just extend from `ContentFileWriteBuilderImpl` directly without the private static impl class that double inherite.
```
private static class DataFileWriteBuilder<D, S>
extends ContentFileWriteBuilderImpl<DataWriteBuilder<D, S>, D, S>
implements DataWriteBuilder<D, S> {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2776215471
##########
core/src/main/java/org/apache/iceberg/formats/FileWriterBuilder.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriter;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <W> the concrete writer type the builder produces
+ * @param <S> the type of the schema for the input data
+ */
+public interface FileWriterBuilder<W extends FileWriter<?, ?>, S> {
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> meta(String property, String value);
+
+ /**
+ * Add the new properties to file metadata for the created file.
+ *
+ * @param properties a map of file metadata properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ FileWriterBuilder<W, S> metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ FileWriterBuilder<W, S> overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the writer
+ * does not support encryption, then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withAADPrefix(ByteBuffer aadPrefix);
+
+ /** Sets the partition specification for the Iceberg metadata. */
+ FileWriterBuilder<W, S> spec(PartitionSpec newSpec);
+
+ /** Sets the partition value for the Iceberg metadata. */
+ FileWriterBuilder<W, S> partition(StructLike partition);
+
+ /** Sets the encryption key metadata for Iceberg metadata. */
+ FileWriterBuilder<W, S> keyMetadata(EncryptionKeyMetadata keyMetadata);
+
+ /** Sets the sort order for the Iceberg metadata. */
+ FileWriterBuilder<W, S> sortOrder(SortOrder sortOrder);
+
+ /** Set the file schema. */
+ FileWriterBuilder<W, S> schema(Schema schema);
+
+ /**
+ * Sets the engine's representation accepted by the writer.
+ *
+ * <p>Some data types require additional type information from the engine schema that cannot be
+ * fully expressed by the Iceberg schema or the data itself. For example, a variant type may use a
+ * shredded representation that relies on engine-specific metadata to map back to the Iceberg
+ * schema.
+ *
+ * <p>The engine schema must be aligned with the Iceberg schema, but may include representation
+ * details that Iceberg considers equivalent.
Review Comment:
I think it's fine here. Engine schema is really a contract between the engine and its registered object model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775986509
##########
core/src/main/java/org/apache/iceberg/formats/BaseFormatModel.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.Map;
+import org.apache.iceberg.Schema;
+
+/**
+ * Base implementation of {@link FormatModel} that provides common functionality for format models.
+ *
+ * <p>This abstract class serves as a foundation for creating format-specific models that handle
+ * reading and writing data in various file formats.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ * @param <W> the writer type produced by the writer function
+ * @param <R> the reader type produced by the reader function
+ * @param <F> the file schema type used by the underlying file format
+ */
+public abstract class BaseFormatModel<D, S, W, R, F> implements FormatModel<D, S> {
+ private final Class<? extends D> type;
+ private final Class<S> schemaType;
+ private final WriterFunction<W, S, F> writerFunction;
+ private final ReaderFunction<R, S, F> readerFunction;
+
+ /**
+ * Constructs a new BaseFormatModel with the specified configuration.
+ *
+ * @param type the row type class for the object model implementation processed by this factory.
+ * @param schemaType the schema type class for the object model implementation processed by this
+ * factory.
+ * @param writerFunction the function used to create writers for this format
+ * @param readerFunction the function used to create readers for this format
+ */
+ protected BaseFormatModel(
+ Class<? extends D> type,
+ Class<S> schemaType,
+ WriterFunction<W, S, F> writerFunction,
+ ReaderFunction<R, S, F> readerFunction) {
+ this.type = type;
+ this.schemaType = schemaType;
+ this.writerFunction = writerFunction;
+ this.readerFunction = readerFunction;
+ }
+
+ @Override
+ public Class<? extends D> type() {
+ return type;
+ }
+
+ @Override
+ public Class<S> schemaType() {
+ return schemaType;
+ }
+
+ /**
+ * Returns the writer function used to create writers for this format.
+ *
+ * @return the writer function
+ */
+ protected WriterFunction<W, S, F> writerFunction() {
+ return writerFunction;
+ }
+
+ /**
+ * Returns the reader function used to create readers for this format.
+ *
+ * @return the reader function
+ */
+ protected ReaderFunction<R, S, F> readerFunction() {
+ return readerFunction;
+ }
+
+ /**
+ * A functional interface for creating writers that can write data in a specific format.
+ *
+ * @param <W> the writer type to be created
+ * @param <S> the type of the schema for the input data
Review Comment:
Added engine specific here, and added the optional to the actual param
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2440237407
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, ?> factory = factoryForPositionDelete(format);
+ WriteBuilder writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @VisibleForTesting
+ static Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> models() {
+ return MODELS;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
+ FormatModel<D, S> model = ((FormatModel<D, S>) MODELS.get(Pair.of(format, type)));
+ Preconditions.checkNotNull(model, "Format model is not registered for %s and %s", format, type);
Review Comment:
nit: I think throwing an IAE here is clearer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2463938966
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Return the row type class for the object model implementation processed by this factory.
+ *
+ * <p>The model types act as a contract specifying the expected data structures for both reading
+ * (converting file formats into output objects) and writing (converting input objects into file
+ * formats). This ensures proper integration between Iceberg's storage layer and processing
+ * engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return the type of the data structures handled by this model implementation
+ */
+ Class<D> type();
+
+ Class<S> schemaType();
+
+ /**
+ * Creates a writer builder for data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder
+ */
+ WriteBuilder writeBuilder(OutputFile outputFile);
Review Comment:
We definitely don't want to expose the internal file format types handled by the FileFormats themselves. We might be able to expose the accepted data types, and the engine specific schemas if we decide to do so. See: https://github.com/apache/iceberg/pull/12774#discussion_r2449364546
Based on the decision there, it could be:
- For Spark: `WriteBuilder<InternalRow, SchemaType>`, `WriteBuilder<InternalRow>`, or WriteBuilder
- For Flink: `WriteBuilder<RowData, RowType>`, `WriteBuilder<RowData>`, or WriteBuilder
- For Generic: `WriteBuilder<Record, Schema>`, `WriteBuilder<Record>`, or WriteBuilder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2438788058
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
+ throw new UnsupportedOperationException("Not implemented yet");
+ }
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ ReadBuilder caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
Review Comment:
nit: "The default implementation sets the filter to be case-sensitive." Isn't this rather relevant for the `caseSensitive` function?
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
+ throw new UnsupportedOperationException("Not implemented yet");
+ }
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ ReadBuilder caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param filter the filter to set
+ */
+ ReadBuilder filter(Expression filter);
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ ReadBuilder set(String key, String value);
Review Comment:
nit: The similar function in WriteBuilder has a more structured comment. Would be nice to follow that pattern in this class too.
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder meta(String property, String value);
+
+ /**
+ * Sets multiple file metadata properties in the created file.
+ *
+ * @param properties file metadata properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /**
+ * Based on the target file content the generated {@link FileAppender} needs different
+ * configuration.
+ */
+ WriteBuilder content(FileContent content);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ WriteBuilder metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ WriteBuilder overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ WriteBuilder withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the reader
Review Comment:
nit: "If the reader does not support encryption" -> "If the writer..."
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
Review Comment:
nit: `isEqual` (without the s) or simply `equals`?
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
Review Comment:
Does this mean that in order to have a format model for other than what is covered by FileFormat now it's required to extend the FileFormat enum too?
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, ?> factory = factoryForPositionDelete(format);
+ WriteBuilder writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @VisibleForTesting
+ static Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> models() {
+ return MODELS;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
+ FormatModel<D, S> model = ((FormatModel<D, S>) MODELS.get(Pair.of(format, type)));
+ Preconditions.checkArgument(
+ model != null, "Format model is not registered for %s and %s", format, type);
+ return model;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static FormatModel<PositionDelete<?>, ?> factoryForPositionDelete(FileFormat format) {
Review Comment:
Do you need a separate factory function for pos deletes? Can't you use the other `factoryFor` method passing PositionDelete.class as arg and do the return type conversion at the calcite?
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
Review Comment:
`The builder supports
* configuration options like schema projection, predicate pushdown, batch size and encryption.`
nit: for me this seems relevant for the ReadBuilder class and probably not here.
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
Review Comment:
nit: ReadBuilder comment mentions that unknown config is ignored. Should we mention it here (and for setAll) too?
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
Review Comment:
nit: some comment about the params would be helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2255015055
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
Review Comment:
Is `B` exposed to callers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254946643
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.dataBuilder(outputFile.encryptingOutputFile());
Review Comment:
So this uses a data builder and just ignores the partition/spec and other data builder methods? If this returns a specific class, then this expose those additional configuration methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254939488
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
Review Comment:
We typically put these at the very top of the class to avoid not knowing where to put it. Otherwise it tends to end up in the middle of static methods or other content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2457758065
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
Review Comment:
Great that we have consensus that the new `WriteBuilder` interface can have generic parameters.
The `Parquet.WriteBuilder` would implement this `WriteBuilder` interface.
```
public static class WriteBuilder
implements InternalData.WriteBuilder, org.apache.iceberg.formats.WriteBuilder<Object, Object> {
```
Peter's earlier attempt is to use `Object` for the generic params to avoid breaking the source compatibility for the existing `Parquet.WriteBuilder`. Don't know if we have agreed on how to move forward for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2415768720
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
Review Comment:
Following `FormatModel`'s definition, should this be a generic parameter like:
```
ReadBuilder outputSchema(S schema)
```
Also, in which case this is useful? I thought `project` method was enough.
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
Review Comment:
I think it's safer to compare class instances, e.g. `model1.getClass() == model2.getClass()`. If they are from different class loaders, we should treat them as different class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2447602244
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
Review Comment:
This feature will be valuable for Variant shredding. When the engine expects a shredded output for a Variant column, it needs to define the expected schema. The reader builder could leverage this information to create a more efficient reader.
In theory, it could also enable generating a DataFrame where a column is defined as long, even if the underlying data file stores int values.
That said, since this functionality is not currently used in the code and based on the discussion [here](https://github.com/apache/iceberg/pull/12774#discussion_r2441349336), I will remove the method for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444383778
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder meta(String property, String value);
+
+ /**
+ * Sets multiple file metadata properties in the created file.
+ *
+ * @param properties file metadata properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /**
+ * Based on the target file content the generated {@link FileAppender} needs different
+ * configuration.
+ */
+ WriteBuilder content(FileContent content);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ WriteBuilder metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ WriteBuilder overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ WriteBuilder withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the reader
+ * does not support encryption, then an exception should be thrown.
+ */
+ WriteBuilder withAADPrefix(ByteBuffer aadPrefix);
+
+ /** Finalizes the configuration and builds the {@link FileAppender}. */
+ <D> FileAppender<D> build() throws IOException;
Review Comment:
We need this, as the FileFormat will still need to provide the FileAppender. It will not be exposed to the end users, but the `DataWriteBuilder`, `EqualityDeleteWriteBuilder` and the `PositionDeleteWriteBuilder` implementations will use it under the hood.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775162464
##########
core/src/main/java/org/apache/iceberg/formats/BaseFormatModel.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.Map;
+import org.apache.iceberg.Schema;
+
+/**
+ * Base implementation of {@link FormatModel} that provides common functionality for format models.
+ *
+ * <p>This abstract class serves as a foundation for creating format-specific models that handle
+ * reading and writing data in various file formats.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ * @param <W> the writer type produced by the writer function
+ * @param <R> the reader type produced by the reader function
+ * @param <F> the file schema type used by the underlying file format
+ */
+public abstract class BaseFormatModel<D, S, W, R, F> implements FormatModel<D, S> {
+ private final Class<? extends D> type;
+ private final Class<S> schemaType;
+ private final WriterFunction<W, S, F> writerFunction;
+ private final ReaderFunction<R, S, F> readerFunction;
+
+ /**
+ * Constructs a new BaseFormatModel with the specified configuration.
+ *
+ * @param type the row type class for the object model implementation processed by this factory.
+ * @param schemaType the schema type class for the object model implementation processed by this
+ * factory.
+ * @param writerFunction the function used to create writers for this format
+ * @param readerFunction the function used to create readers for this format
+ */
+ protected BaseFormatModel(
+ Class<? extends D> type,
+ Class<S> schemaType,
+ WriterFunction<W, S, F> writerFunction,
+ ReaderFunction<R, S, F> readerFunction) {
+ this.type = type;
+ this.schemaType = schemaType;
+ this.writerFunction = writerFunction;
+ this.readerFunction = readerFunction;
+ }
+
+ @Override
+ public Class<? extends D> type() {
+ return type;
+ }
+
+ @Override
+ public Class<S> schemaType() {
+ return schemaType;
+ }
+
+ /**
+ * Returns the writer function used to create writers for this format.
+ *
+ * @return the writer function
+ */
+ protected WriterFunction<W, S, F> writerFunction() {
+ return writerFunction;
+ }
+
+ /**
+ * Returns the reader function used to create readers for this format.
+ *
+ * @return the reader function
+ */
+ protected ReaderFunction<R, S, F> readerFunction() {
+ return readerFunction;
+ }
+
+ /**
+ * A functional interface for creating writers that can write data in a specific format.
+ *
+ * @param <W> the writer type to be created
+ * @param <S> the type of the schema for the input data
+ * @param <F> the file schema type used by the underlying file format
+ */
+ @FunctionalInterface
+ public interface WriterFunction<W, S, F> {
+ /**
+ * Creates a writer for the given schemas.
+ *
+ * @param icebergSchema the Iceberg schema defining the table structure
+ * @param fileSchema the file format specific target schema for the output files
+ * @param engineSchema the engine specific schema for the input data
+ * @return a writer configured for the given schemas
+ */
+ W write(Schema icebergSchema, F fileSchema, S engineSchema);
+ }
+
+ /**
+ * A functional interface for creating readers that can read data from a specific format.
+ *
+ * @param <R> the reader type to be created
+ * @param <S> the type of the schema for the output data
+ * @param <F> the file schema type used by the underlying file format
+ */
+ @FunctionalInterface
+ public interface ReaderFunction<R, S, F> {
+ /**
+ * Creates a reader for the given schemas.
+ *
+ * @param icebergSchema the Iceberg schema defining the table structure
+ * @param fileSchema the file format specific source schema for the input files
Review Comment:
I think it would be more clear to handle the type information in the docs for `F` and describe where this schema comes from here. This is the schema that the file was written with, although it can be `null` in the case of Avro files because it is passed in later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#issuecomment-3865521713
Opened the follow-up PRs:
- Parquet: #15253
- Avro: #15254
- ORC: #15255
- Javadoc tweaks: #15257
I would appreciate reviews on them.
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2066013312
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which should be implemented by the data file format implementations. The {@link
+ * AppenderBuilder} will be parametrized based on the user provided configuration and finally the
+ * {@link AppenderBuilder#build(AppenderBuilder.WriteMode)} method is used to generate the appender
+ * for the specific writer use-cases. The following input should be handled by the appender in the
+ * specific modes:
+ *
+ * <ul>
+ * <li>The appender's engine specific input type
+ * <ul>
+ * <li>{@link AppenderBuilder.WriteMode#APPENDER}
+ * <li>{@link AppenderBuilder.WriteMode#DATA_WRITER}
+ * <li>{@link AppenderBuilder.WriteMode#EQUALITY_DELETE_WRITER}
+ * </ul>
+ * <li>{@link org.apache.iceberg.deletes.PositionDelete} where the type of the row is the
+ * appender's engine specific input type
+ * <ul>
+ * <li>{@link AppenderBuilder.WriteMode#POSITION_DELETE_WRITER}
+ * <li>{@link AppenderBuilder.WriteMode#POSITION_DELETE_WITH_ROW_WRITER}
+ * </ul>
+ * </ul>
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If encryption is not supported by the reader
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data prefix used for writing the file. If encryption is not
+ * supported by the reader then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the engine native schema for the input. Defines the input type when there is N to 1
+ * mapping between the engine type and the Iceberg type, and providing the Iceberg schema is not
+ * enough for the conversion.
+ */
+ B engineSchema(E newEngineSchema);
+
+ /**
+ * Builds the {@link FileAppender} for the configured {@link WriteMode}. Could change several
+ * use-case specific configurations, like:
+ *
+ * <ul>
+ * <li>Mode specific writer context (typically different for data and delete files).
+ * <li>Writer functions to accept data rows, or {@link
+ * org.apache.iceberg.deletes.PositionDelete}s
+ * </ul>
+ */
+ <D> FileAppender<D> build(WriteMode mode) throws IOException;
+
+ /**
+ * Writer modes. Based on the mode {@link #build(WriteMode)} could alter the appender
+ * configuration when creating the {@link FileAppender}.
+ */
+ enum WriteMode {
+ /** Mode for appending data to a file. */
+ APPENDER,
Review Comment:
Good point. I included it since I introduced a new mode for every writer mode, but they are always the same.
I will remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2066023603
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
Review Comment:
It was concurrent already, but we used only the API provided by the `Map`.
But agree, that it is better to explicitly define as concurrent.
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2075842898
##########
data/src/main/java/org/apache/iceberg/data/AppenderBuilder.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder for generating a {@link FileAppender}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E>
Review Comment:
Good news!
I have found a way to relatively simply remove the `data.AppenderBuilder`. I needed to add `WriteMode mode` parameter to the `ObjectModel.appenderBuilder(OutputFile outputFile, WriteMode mode);`, but I think this is an acceptable tradeoff.
@liurenjie1024, @jbonofre could you please double check?
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2079085605
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
Review Comment:
There are several public APIs where the contract requires us to keep the current `overwrite(boolean)`. These APIs will be deprecated, but in 1.10 we still need to provide them. OTOH we are planning to use the new API internally in every case, so as a temporary measure we will provide `overwrite(boolean)`. When the old APIs are removed, we will not need the new method in the new API, so it could be removed and streamlined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2079076379
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
As an end result, we need to provide 4 interfaces from the ObjectModel:
- Appender - generates a `FileAppender` which just writes the data to an output file
- DataWriter - generates a `DataWriter` which generates DataFiles
- PositionDeleteWriter - generates a `PositionDeleteWriter` which generates DeleteFile
- EqualityDeleteWriter - generates a `EqualityDeleteWriter` which generates DeleteFile
The PR doesn't touch the results (`FileAppender`, `DataWriter`, `PositionDeleteWriter`, `EqualityDeleteWriter`). I think it is better to keep these names so it is easier to understand what will be the result
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095395202
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
We have a matrix between the `File Format` X `Engine Internal Object Model`.
I don't really like the `DataIOFactory` as it doesn't contain any of the `File Format` or `Engine Internal Object Model`, but agree with you that even `ObjectModel` is a bit odd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2099792831
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
Review Comment:
There are several public APIs where the contract requires us to keep the current `overwrite(boolean)`. These APIs will be deprecated, but in 1.10 we still need to provide them. OTOH we are planning to use the new API internally in every case, so as a temporary measure we will provide `overwrite(boolean)`. When the old APIs are removed, we will not need the new method in the new API, so it could be removed and streamlined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2099787997
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
Review Comment:
There are 2 main things:
- WriterContext - the writer needs the info if it is a data or a delete file we are writing. Based on this, different table properties are used, like `write.delete.parquet.row-group-size-bytes` or `write.parquet.row-group-size-bytes`
- The logic of writing the positional deletes are dependent on the file format `PositionDeleteStructWriter` for Parquet, `PositionAndRowDatumWriter` for Avro, `GenericOrcWriters.positionDelete` for ORC. They need different handling/parametrization for different file formats, which I wanted to keep internal to the file formats themselves. But then they need the info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#issuecomment-2897962911
> @huaxingao, @pvary Could you take a look from a comet prospective? I know you have some custom code that would be using this as well
Originally I thought that the `comet` could be just another `FileAccessFactory` to register, but based on @rdblue's suggestion I have merged it back to the spark-vectorized/Parquet factory. If the current API is not working for the custom code, that direction could be something which might be worth to explore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163712626
##########
api/src/main/java/org/apache/iceberg/io/FileReader.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.util.Map;
+
+/**
+ * A reader capable of reading files and returning records of type {@code D}.
+ *
+ * <p>This interface should be implemented by classes that not only read records from files but also
+ * provide metadata stored in the file.
+ *
+ * @param <D> the record type
+ */
+public interface FileReader<D> extends CloseableIterable<D> {
+ Map<String, String> meta();
Review Comment:
Reverted.
CC: @RussellSpitzer, @danielcweeks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163628556
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user-provided setting if it is not supported by them.
+ */
+ default B reuseContainers() {
Review Comment:
Currently ORC doesn't have this option, and automatically reuses the containers (at least based on some comments). Defaulting to not reusing could be a breaking change. If we accept this then we can change this behavior. Changed the comment, and we can deal with this later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162696513
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ */
+public interface FileAccessFactory<E, D> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
+ *
+ * <p>The object model names (such as "generic", "spark", "spark-vectorized", "flink", "arrow")
+ * act as a contract specifying the expected data structures for both reading (converting file
+ * formats into output objects) and writing (converting input objects into file formats). This
+ * ensures proper integration between Iceberg's storage layer and processing engines.
+ *
+ * @return string identifier for this object model implementation
+ */
+ String objectModelName();
+
+ /**
+ * Creates a writer builder for standard data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory for regular data content.
+ *
+ * <p>The builder follows the fluent pattern for configuring writer properties like compression,
+ * encryption, row group size, and other format-specific options.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder for standard data files
+ * @param <B> the concrete builder type for method chaining
+ */
+ <B extends WriteBuilder<B, E, D>> B dataWriteBuilder(OutputFile outputFile);
Review Comment:
The existing builders are created using `writeData` and `writeDeletes`. Why are the names longer here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2098593221
##########
data/src/main/java/org/apache/iceberg/data/PositionDeleteWriteBuilder.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * A specialized builder for creating position-based delete file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link PositionDeleteWriter} instances.
+ *
+ * <p>The builder provides methods to configure the schema for the row data that might be included
+ * with the position deletes through {@link #rowSchema(Schema)}, enabling optional preservation of
+ * deleted record content.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type required by the writer for data conversion
+ */
+public interface PositionDeleteWriteBuilder<B extends PositionDeleteWriteBuilder<B, E>, E>
+ extends ContentFileWriteBuilder<B, E> {
+ /** Sets the row schema for the delete writers. */
+ B rowSchema(Schema newSchema);
Review Comment:
```
* <p>If {@link #rowSchema(Schema)} is configured, the position delete records should include the
* content of the deleted rows. These row values should match the engine schema specified via
* {@link #dataSchema(Object)} and will be converted to the target Iceberg schema defined by
* {@link #rowSchema(Schema)}.
```
Maybe move the Javadoc from the method below to this method. do we just need a boolean flag? otherwise, we need to validate that `rowSchema` matches `dataSchema`.
I guess this is also obsolete with the V3 delete vector where content of the deleted rows are never included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2146066061
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user-provided setting if it is not supported by them.
+ */
+ default B reuseContainers() {
+ // Skip container reuse configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata and not coming from the data files themselves. The keys of the map are the column
+ * ids, the values are the accessors generating the values.
+ */
+ B constantFieldAccessors(Map<Integer, ?> constantFieldAccessors);
+
+ /** Sets a mapping from external schema names to Iceberg type IDs. */
+ B nameMapping(NameMapping newNameMapping);
+
+ /**
+ * Sets the file encryption key used for reading the file. If the reader does not support
+ * encryption, then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (AAD) prefix for decryption. If the reader does not
+ * support decryption, then an exception should be thrown.
+ */
+ default B fileAADPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /** Builds the reader. */
+ CloseableIterable<D> build();
Review Comment:
Something I've been running into is that the CloseableIterator here means that we can build a reader, but can't access the file metadata if it exists. I'm wondering if this should be shaped a little differently, May CloseableIterator<D> with an HasMetadata or something interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2150013016
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user-provided setting if it is not supported by them.
+ */
+ default B reuseContainers() {
+ // Skip container reuse configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata and not coming from the data files themselves. The keys of the map are the column
+ * ids, the values are the accessors generating the values.
+ */
+ B constantFieldAccessors(Map<Integer, ?> constantFieldAccessors);
+
+ /** Sets a mapping from external schema names to Iceberg type IDs. */
+ B nameMapping(NameMapping newNameMapping);
+
+ /**
+ * Sets the file encryption key used for reading the file. If the reader does not support
+ * encryption, then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (AAD) prefix for decryption. If the reader does not
+ * support decryption, then an exception should be thrown.
+ */
+ default B fileAADPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /** Builds the reader. */
+ CloseableIterable<D> build();
Review Comment:
Ok. This is the 3rd, or 4th time that this one was requested:
- https://github.com/apache/iceberg/pull/12298#discussion_r2038212300 - by @danielcweeks
- In our Slack discussions by @stevenzwu
- IIRC even @liurenjie1024 suggested something like this.
I have originally pushed back on this based on @rdblue's and @danielcweeks's previous discussion on one of the `InternalData` PRs: https://github.com/apache/iceberg/pull/12476#discussion_r1985473714
Adding a separate interface will help in future extensibility, so after getting so much heat on this, I added a new interface and started to use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228030091
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user-provided setting if it is not supported by them.
+ */
+ default B reuseContainers() {
+ // Skip container reuse configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata and not coming from the data files themselves. The keys of the map are the column
+ * ids, the values are the accessors generating the values.
+ */
+ B constantFieldAccessors(Map<Integer, ?> constantFieldAccessors);
+
+ /** Sets a mapping from external schema names to Iceberg type IDs. */
+ B nameMapping(NameMapping newNameMapping);
+
+ /**
+ * Sets the file encryption key used for reading the file. If the reader does not support
+ * encryption, then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (AAD) prefix for decryption. If the reader does not
+ * support decryption, then an exception should be thrown.
+ */
+ default B fileAADPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /** Builds the reader. */
+ CloseableIterable<D> build();
Review Comment:
I got a message offline, that we will not handle metadata reads for now. As a general rule, we should not rely on metadata to fetch information about the files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2258454786
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
Review Comment:
Also, the WriteBuilder API already has a `build` method which returns the appender, which we don't want to expose here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2097304773
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
If we rename all of the `FileAppender` occurrences to `FileWriter` we need to modify 130 files (not checked every file, but seems correct by randomly checking files). Seems a bit excessive to me.
We can still decide to call the class WriteBuilder and return FileAppender (this is how it is done in InternalData)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2727090462
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static synchronized void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null,
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @param <S> the type of the output schema for the reader
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<? extends D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<? extends D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return CommonWriteBuilderImpl.forDataFile(
+ factory.writeBuilder(outputFile), outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return CommonWriteBuilderImpl.forEqualityDelete(
+ factory.writeBuilder(outputFile), outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ @SuppressWarnings("rawtypes")
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete, ?> factory = factoryFor(format, PositionDelete.class);
+ return CommonWriteBuilderImpl.forPositionDelete(
+ factory.writeBuilder(outputFile), outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @VisibleForTesting
+ static Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> models() {
+ return MODELS;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<? extends D> type) {
+ FormatModel<D, S> model = (FormatModel<D, S>) MODELS.get(Pair.of(format, type));
+ Preconditions.checkArgument(
+ model != null, "Format model is not registered for format %s and type %s", format, type);
+ return model;
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
Review Comment:
This is not straightforward. The models invoke a static register method and do not return a collection of registered models:
```
public static synchronized void register(FormatModel<?, ?> formatModel)
```
As a result, we have no direct access to the registered model instances themselves.
Additionally, static methods are inherently difficult to test, and this is something we likely worth the effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095229232
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s, {@link AppenderBuilder}s and writer
+ * builders ({@link DataWriteBuilder}, {@link EqualityDeleteWriteBuilder}, {@link
+ * PositionDeleteWriteBuilder}). Based on the `file format` and the requested `object model name`
+ * the registry returns the correct reader and writer builders. These builders could be used to
+ * generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones either returned directly or wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
Review Comment:
If I remember correctly @danielcweeks had a comment suggesting to move as much as possible to `data`. I agree with him, as I feel that the `core` module starts to get bloated, and everything is pushed there.
The `data` module already contains classes for accessing data independently of the actual File Format. I think the ObjectModelRegistry has the same function.
Another possibility to separate the classes based on the user/implementer.
The `ObjectModel`, `AppenderBuilder`, `ReadBuilder` are implemented by the File Formats.
The `ObjectModelRegistry`, `DataWriteBuilder`, `EqualityDeleteWriteBuilder`, `PositionDeleteWriteBuilder` are used by the engines. The `AppenderBuilder`, `ReadBuilder` is a bit outlier here, as they are exposed directly, but I have kept them in the `core` based on the "ownership".
So all-in-all, I prefer this separation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095462749
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * The name of the object model. Allows users to specify the object model to map the data file for
+ * reading and writing.
+ */
+ String name();
+
+ /**
+ * The appender builder for the output file which writes the data in the specified file format and
+ * accepts the records defined by this object model. The 'mode' parameter defines the input type
+ * for the specific writer use-cases. The appender should handle the following input in the
+ * specific modes:
+ *
+ * <ul>
+ * <li>The appender's engine specific input type
+ * <ul>
+ * <li>{@link WriteMode#DATA_WRITER}
+ * <li>{@link WriteMode#EQUALITY_DELETE_WRITER}
+ * </ul>
+ * <li>{@link org.apache.iceberg.deletes.PositionDelete} where the type of the row is the
+ * appender's engine specific input type when the 'mode' is {@link
+ * WriteMode#POSITION_DELETE_WRITER}
+ * </ul>
+ *
+ * @param outputFile to write to
+ * @param mode for the appender
+ * @return the appender builder
+ * @param <B> The type of the appender builder
+ */
+ <B extends AppenderBuilder<B, E>> B appenderBuilder(OutputFile outputFile, WriteMode mode);
Review Comment:
Went ahead and used `FileContent`.
About the 3 different method approach:
- We would like to expose the same Appender interface for Flink (or other users) that it is provided by the File Format. There we don't want any mention of "FileContent" or "WriteMode" or multiple builder methods.
That is why I opted for passing the mode in the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2096253249
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
if we can separate out the file format handling and the engine in-memory object model, then this class (file format specific) can be called `FileFormatIOFactory` or `FileAccessFactory`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2093651265
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
I see. It is odd that the symmetric counter part of `ReadBuilder` is not `WriteBuilder`.
our current class naming is a bit inconsistent in terms of `Writer` vs `Appender`. E.g. we have `ParquetWriter implements FileAppender`. In my mind, `FileAppender` should be named as `FileWriter` for write data to a specific file format. And the current `FileWriter` should be called `ContentFileWriter`, because `ContentFile` is a data, position delete, or equality delete file. But I guess we might be too late for that.
I am also not sure why `FileAppender` is part of iceberg-api module. It is not used in `iceberg-api`. `iceberg-core` is probably a better home.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2100280432
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If the reader does not support encryption,
Review Comment:
Thanks for catching. Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2340489407
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()} ()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+ Preconditions.checkArgument(
Review Comment:
This was a specific requirement from @danielcweeks. I don't have a strong opinion here. If we have consensus in any direction, I happy to follow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2340498445
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.DeleteSchemaUtil;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the type of data records the writer will accept
+ */
+@SuppressWarnings("unchecked")
+abstract class ContentFileWriteBuilderImpl<B extends ContentFileWriteBuilder<B, S>, D, S>
+ implements ContentFileWriteBuilder<B, S> {
+ private final WriteBuilder<D, S> writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ static <D, S> DataWriteBuilder<D, S> forDataFile(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new DataFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> EqualityDeleteWriteBuilder<D, S> forEqualityDelete(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new EqualityDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> PositionDeleteWriteBuilder<D, S> forPositionDelete(
+ WriteBuilder<D, S> writeBuilder,
+ String location,
+ FileFormat format,
+ Function<Schema, Function<PositionDelete<D>, D>> positionDeleteConverter) {
+ return new PositionDeleteFileWriteBuilder<>(
+ writeBuilder, location, format, positionDeleteConverter);
+ }
+
+ private ContentFileWriteBuilderImpl(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public B schema(Schema schema) {
+ writeBuilder.schema(schema);
+ return self();
+ }
+
+ @Override
+ public B inputSchema(S schema) {
+ writeBuilder.inputSchema(schema);
+ return self();
+ }
+
+ @Override
+ public B set(String property, String value) {
+ writeBuilder.set(property, value);
+ return self();
+ }
+
+ @Override
+ public B meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return self();
+ }
+
+ @Override
+ public B metricsConfig(MetricsConfig metricsConfig) {
+ writeBuilder.metricsConfig(metricsConfig);
+ return self();
+ }
+
+ @Override
+ public B overwrite() {
+ writeBuilder.overwrite();
+ return self();
+ }
+
+ @Override
+ public B fileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.fileEncryptionKey(encryptionKey);
+ return self();
+ }
+
+ @Override
+ public B fileAADPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.fileAADPrefix(aadPrefix);
+ return self();
+ }
+
+ @Override
+ public B spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return self();
+ }
+
+ @Override
+ public B partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return self();
+ }
+
+ @Override
+ public B keyMetadata(EncryptionKeyMetadata newKeyMetadata) {
+ this.keyMetadata = newKeyMetadata;
+ return self();
+ }
+
+ @Override
+ public B sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return self();
+ }
+
+ private static class DataFileWriteBuilder<D, S>
+ extends ContentFileWriteBuilderImpl<DataWriteBuilder<D, S>, D, S>
+ implements DataWriteBuilder<D, S> {
+ private DataFileWriteBuilder(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ super(writeBuilder, location, format);
+ }
+
+ @Override
+ public DataFileWriteBuilder<D, S> self() {
+ return this;
+ }
+
+ @Override
+ public DataWriter<D> build() throws IOException {
+ Preconditions.checkArgument(super.spec != null, "Cannot create data writer without spec");
+ Preconditions.checkArgument(
+ super.spec.isUnpartitioned() || super.partition != null,
+ "Partition must not be null when creating data writer for partitioned spec");
+
+ return new DataWriter<>(
+ super.writeBuilder.build(),
+ super.format,
+ super.location,
+ super.spec,
+ super.partition,
+ super.keyMetadata,
+ super.sortOrder);
+ }
+ }
+
+ private static class EqualityDeleteFileWriteBuilder<D, S>
+ extends ContentFileWriteBuilderImpl<EqualityDeleteWriteBuilder<D, S>, D, S>
+ implements EqualityDeleteWriteBuilder<D, S> {
+ private Schema rowSchema = null;
+ private int[] equalityFieldIds = null;
+
+ private EqualityDeleteFileWriteBuilder(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ super(writeBuilder, location, format);
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> self() {
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> rowSchema(Schema schema) {
+ this.rowSchema = schema;
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> equalityFieldIds(int... fieldIds) {
+ this.equalityFieldIds = fieldIds;
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteWriter<D> build() throws IOException {
+ Preconditions.checkState(
+ rowSchema != null, "Cannot create equality delete file without a schema");
+ Preconditions.checkState(
+ equalityFieldIds != null, "Cannot create equality delete file without delete field ids");
+ Preconditions.checkArgument(
+ super.spec != null, "Spec must not be null when creating equality delete writer");
+ Preconditions.checkArgument(
+ super.spec.isUnpartitioned() || super.partition != null,
+ "Partition must not be null for partitioned writes");
+
+ return new EqualityDeleteWriter<>(
+ super.writeBuilder
+ .schema(rowSchema)
+ .meta("delete-type", "equality")
+ .meta(
+ "delete-field-ids",
+ IntStream.of(equalityFieldIds)
+ .mapToObj(Objects::toString)
+ .collect(Collectors.joining(", ")))
+ .build(),
+ super.format,
+ super.location,
+ super.spec,
+ super.partition,
+ super.keyMetadata,
+ super.sortOrder,
+ equalityFieldIds);
+ }
+ }
+
+ private static class PositionDeleteFileWriteBuilder<D, S>
+ extends ContentFileWriteBuilderImpl<PositionDeleteWriteBuilder<D, S>, D, S>
+ implements PositionDeleteWriteBuilder<D, S> {
+ private final Function<Schema, Function<PositionDelete<D>, D>> positionDeleteConverter;
+ private Schema rowSchema = null;
+
+ private PositionDeleteFileWriteBuilder(
+ WriteBuilder<D, S> writeBuilder,
+ String location,
+ FileFormat format,
+ Function<Schema, Function<PositionDelete<D>, D>> positionDeleteConverter) {
+ super(writeBuilder, location, format);
+ this.positionDeleteConverter = positionDeleteConverter;
+ }
+
+ @Override
+ public PositionDeleteFileWriteBuilder<D, S> self() {
+ return this;
+ }
+
+ @Override
+ public PositionDeleteFileWriteBuilder<D, S> rowSchema(Schema schema) {
+ this.rowSchema = schema;
+ return this;
+ }
+
+ @Override
+ public PositionDeleteWriter<D> build() throws IOException {
+ Preconditions.checkArgument(
+ positionDeleteConverter != null,
+ "Positional delete converter must not be null when creating position delete writer");
+ Preconditions.checkArgument(
+ super.spec != null, "Spec must not be null when creating position delete writer");
+ Preconditions.checkArgument(
+ super.spec.isUnpartitioned() || super.partition != null,
+ "Partition must not be null for partitioned writes");
+
+ return new PositionDeleteWriter<D>(
+ (FileAppender)
+ new ConvertingFileAppender<>(
+ super.writeBuilder
+ .meta("delete-type", "position")
+ .schema(DeleteSchemaUtil.posDeleteSchema(rowSchema))
+ .build(),
+ positionDeleteConverter.apply(rowSchema),
+ rowSchema),
+ super.format,
+ super.location,
+ super.spec,
+ super.partition,
+ super.keyMetadata);
+ }
+ }
+
+ private static class ConvertingFileAppender<D> implements FileAppender<PositionDelete<D>> {
Review Comment:
Changed to `PositionDeleteFileAppender` - that seems more appropriate
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.DeleteSchemaUtil;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the type of data records the writer will accept
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228050586
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ B project(Schema newSchema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param newCaseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean newCaseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ B filter(Expression newFilter);
Review Comment:
removed `new` from every parameter
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #registerFormatModel(FormatModel)}
+ * and used for creating readers and writers. Read builders are returned directly from the factory.
+ * Write builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerFormatModel(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serves %s",
Review Comment:
Thanks, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228062346
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #registerFormatModel(FormatModel)}
+ * and used for creating readers and writers. Read builders are returned directly from the factory.
+ * Write builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerFormatModel(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serves %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
Review Comment:
The `FormatModelRegistry` uses the same patterns as the `InternalData`, and decided to stick to the same patterns here whenever it is possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228093122
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param start the start position for this read
+ * @param length the length of the range this read should scan
+ */
+ B split(long start, long length);
+
+ /** Set the projection schema. */
+ B project(Schema schema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param saseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean saseSensitive);
Review Comment:
Ehh š¢
Sorry for the typo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254989845
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
Review Comment:
I think there is probably a better name than `ContentFileWriteBuilder`. The term `ContentFile` is just a generic internal name that we use as a base for both `DataFile` and `DeleteFile` and it isn't a well-defined concept in the format. Leaking it into exposed APIs like this is not a good use. We are also going to be refactoring those classes for v4 metadata.
This interface is also a little strange. it looks like it has all of the methods from `WriteBuilder` except for `build`. It adds 4 methods to set metadata that get passed into `DataFile` and `DeleteFile` metadata: `spec`, `partition`, `key_metadata`, and `sortOrder`. I wonder if this could just be an interface with those last 4 that gets added to `DataWriteBuilder` and the delete builders instead?
I know that this is trying to share a lot of code, but the data/pos-deletes/eq-deletes builders already exist in multiple places and don't share code already. So it makes sense to me that those builders could continue to be standalone builders that wrap an appender builder from the format. In that case, maybe an interface with these methods isn't needed at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254951788
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the input data type for the writer
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, D>, D> {
+ /** Set the file schema. */
+ B fileSchema(Schema fileSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ @SuppressWarnings("unchecked")
+ default B set(Map<String, String> properties) {
Review Comment:
Why was this renamed from `setAll` to `set`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257488208
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
+
+ /** Set the file schema. */
+ B fileSchema(Schema fileSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ B set(Map<String, String> properties);
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
+
+ /** Set the file schema. */
+ B fileSchema(Schema fileSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ B set(Map<String, String> properties);
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /**
+ * Add the new properties to file metadata for the created file.
+ *
+ * @param properties a map of file metadata properties
+ * @return this for method chaining
+ */
+ B meta(Map<String, String> properties);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257489928
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2227978908
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #registerFormatModel(FormatModel)}
+ * and used for creating readers and writers. Read builders are returned directly from the factory.
+ * Write builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerFormatModel(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serves %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
Review Comment:
maybe it makes sense to make this class a singleton so that the format models are lazily initialized. With the static initializer all of the init logic happens when the class is first loaded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163718347
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
Review Comment:
I followed the pattern that if something is not mandatory on the API then I provided a default implementation for it.
This is an edge case, as most reader will have some kind of config, so removed the default implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2375657712
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <S> the type of the schema for the input data
+ * @param <D> the type of data records the writer will accept
+ */
+@SuppressWarnings("unchecked")
+abstract class ContentFileWriteBuilderImpl<B extends ContentFileWriteBuilder<B, S>, D, S>
+ implements ContentFileWriteBuilder<B, S> {
+ private final WriteBuilder<D, S> writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ static <D, S> DataWriteBuilder<D, S> forDataFile(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new DataFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> EqualityDeleteWriteBuilder<D, S> forEqualityDelete(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new EqualityDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> PositionDeleteWriteBuilder<D, S> forPositionDelete(
+ WriteBuilder<PositionDelete<D>, S> writeBuilder, String location, FileFormat format) {
+ return new PositionDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ private ContentFileWriteBuilderImpl(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public B set(String property, String value) {
+ writeBuilder.set(property, value);
+ return self();
+ }
+
+ @Override
+ public B meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return self();
+ }
+
+ @Override
+ public B metricsConfig(MetricsConfig metricsConfig) {
+ writeBuilder.metricsConfig(metricsConfig);
+ return self();
+ }
+
+ @Override
+ public B overwrite() {
+ writeBuilder.overwrite();
+ return self();
+ }
+
+ @Override
+ public B fileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.fileEncryptionKey(encryptionKey);
+ return self();
+ }
+
+ @Override
+ public B fileAADPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.fileAADPrefix(aadPrefix);
+ return self();
+ }
+
+ @Override
+ public B spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return self();
+ }
+
+ @Override
+ public B partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return self();
+ }
+
+ @Override
+ public B keyMetadata(EncryptionKeyMetadata newKeyMetadata) {
+ this.keyMetadata = newKeyMetadata;
+ return self();
+ }
+
+ @Override
+ public B sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return self();
+ }
+
+ private static class DataFileWriteBuilder<D, S>
Review Comment:
We have many builder interfaces in this design.
That said, I'm open for other suggestions. Let's let others weight in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2507598334
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
Review Comment:
`register` method uses `get` (line 94)) and `put` (line 104) separately. ConcurrentHahsMap doesn't provide the thread safety.
`putIfAbsent` is also not a good fit here due to the assertion/override behavior. Maybe just make this method `synchronized` on the `MODELS` object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2507626732
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <D> the output data type produced by the reader
+ * @param <S> the type of the schema for the output data type
+ */
+public interface ReadBuilder<D, S> {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder<D, S> split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder<D, S> project(Schema schema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting. The default value is <code>true</code>.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ ReadBuilder<D, S> caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again.
+ *
+ * @param filter the filter to set
+ */
+ ReadBuilder<D, S> filter(Expression filter);
+
+ /**
+ * Set a reader configuration property which affects the reader behavior. Reader builders should
+ * ignore configuration keys not known for them.
+ *
+ * @param key a reader config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ ReadBuilder<D, S> set(String key, String value);
+
+ /**
+ * Sets multiple reader configuration properties that affect the reader behavior. Reader builders
+ * should ignore configuration keys not known for them.
+ *
+ * @param properties reader config properties to set
+ * @return this for method chaining
+ */
+ default ReadBuilder<D, S> setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /** Enables reusing the containers returned by the reader. Decreases pressure on GC. */
+ ReadBuilder<D, S> reuseContainers();
+
+ /** Sets the batch size for vectorized readers. */
+ ReadBuilder<D, S> recordsPerBatch(int numRowsPerBatch);
+
+ /**
+ * Contains the values in the result objects which are coming from metadata and not coming from
+ * the data files themselves. The keys of the map are the column ids, the values are the constant
+ * values to be used in the result.
+ */
+ ReadBuilder<D, S> constantValues(Map<Integer, ?> constantValues);
Review Comment:
nit: can we keep the existing name `idToConstant`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2550885450
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -229,4 +206,27 @@ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> t
model != null, "Format model is not registered for format %s and type %s", format, type);
return model;
}
+
+ private static boolean equals(FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
Review Comment:
Changed the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2771675048
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and {@link
+ * FileWriterBuilder}s for writing various types of Iceberg content files. The appropriate builder
+ * is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private FormatModelRegistry() {}
+
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static synchronized void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null,
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @param <S> the type of the output schema for the reader
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<? extends D> type, InputFile inputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return model.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> FileWriterBuilder<DataWriter<D>, S> dataWriteBuilder(
+ FileFormat format, Class<? extends D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return FileWriterBuilderImpl.forDataFile(model, outputFile);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> FileWriterBuilder<EqualityDeleteWriter<D>, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return FileWriterBuilderImpl.forEqualityDelete(model, outputFile);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public static <D> FileWriterBuilder<PositionDeleteWriter<D>, ?> positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<D>, ?> model =
+ (FormatModel<PositionDelete<D>, ?>) (FormatModel) modelFor(format, PositionDelete.class);
Review Comment:
Looks like the double cast is the cleanest option:
```java
Class<PositionDelete<D>> deleteClass = (Class<PositionDelete<D>>) (Class<?>) PositionDelete.class;
FormatModel<PositionDelete<D>, ?> model = FormatModelRegistry.modelFor(format, deleteClass);
return FileWriterBuilderImpl.forPositionDelete(model, outputFile);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775173612
##########
core/src/main/java/org/apache/iceberg/formats/FileWriterBuilder.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriter;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <W> the concrete writer type the builder produces
+ * @param <S> the type of the schema for the input data
+ */
+public interface FileWriterBuilder<W extends FileWriter<?, ?>, S> {
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> meta(String property, String value);
+
+ /**
+ * Add the new properties to file metadata for the created file.
+ *
+ * @param properties a map of file metadata properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ FileWriterBuilder<W, S> metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ FileWriterBuilder<W, S> overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the writer
+ * does not support encryption, then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withAADPrefix(ByteBuffer aadPrefix);
+
+ /** Sets the partition specification for the Iceberg metadata. */
+ FileWriterBuilder<W, S> spec(PartitionSpec newSpec);
+
+ /** Sets the partition value for the Iceberg metadata. */
+ FileWriterBuilder<W, S> partition(StructLike partition);
+
+ /** Sets the encryption key metadata for Iceberg metadata. */
+ FileWriterBuilder<W, S> keyMetadata(EncryptionKeyMetadata keyMetadata);
+
+ /** Sets the sort order for the Iceberg metadata. */
+ FileWriterBuilder<W, S> sortOrder(SortOrder sortOrder);
+
+ /** Set the file schema. */
+ FileWriterBuilder<W, S> schema(Schema schema);
+
+ /**
+ * Sets the engine's representation accepted by the writer.
+ *
+ * <p>Some data types require additional type information from the engine schema that cannot be
+ * fully expressed by the Iceberg schema or the data itself. For example, a variant type may use a
+ * shredded representation that relies on engine-specific metadata to map back to the Iceberg
+ * schema.
Review Comment:
I think a simple example (`tinyint` / `int`) would help as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775171120
##########
core/src/main/java/org/apache/iceberg/formats/FileWriterBuilder.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriter;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <W> the concrete writer type the builder produces
+ * @param <S> the type of the schema for the input data
+ */
+public interface FileWriterBuilder<W extends FileWriter<?, ?>, S> {
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ FileWriterBuilder<W, S> meta(String property, String value);
+
+ /**
+ * Add the new properties to file metadata for the created file.
+ *
+ * @param properties a map of file metadata properties
+ * @return this for method chaining
+ */
+ default FileWriterBuilder<W, S> meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ FileWriterBuilder<W, S> metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ FileWriterBuilder<W, S> overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the writer
+ * does not support encryption, then an exception should be thrown.
+ */
+ FileWriterBuilder<W, S> withAADPrefix(ByteBuffer aadPrefix);
+
+ /** Sets the partition specification for the Iceberg metadata. */
+ FileWriterBuilder<W, S> spec(PartitionSpec newSpec);
+
+ /** Sets the partition value for the Iceberg metadata. */
+ FileWriterBuilder<W, S> partition(StructLike partition);
+
+ /** Sets the encryption key metadata for Iceberg metadata. */
+ FileWriterBuilder<W, S> keyMetadata(EncryptionKeyMetadata keyMetadata);
+
+ /** Sets the sort order for the Iceberg metadata. */
+ FileWriterBuilder<W, S> sortOrder(SortOrder sortOrder);
+
+ /** Set the file schema. */
+ FileWriterBuilder<W, S> schema(Schema schema);
+
+ /**
+ * Sets the engine's representation accepted by the writer.
Review Comment:
This is a little unclear; I'd prefer "Sets the engine-specific schema that describes records accepted by the writer".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2776224708
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and {@link
+ * FileWriterBuilder}s for writing various types of Iceberg content files. The appropriate builder
+ * is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
Review Comment:
Not sure that this part about the write builders being wrapped needs to be mentioned here. What about "Readers and writers are created using builders from the static factory methods of this class."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2440855791
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
Review Comment:
Maybe?
```
isEquals(FileFormat<?, ?> first, FormatModel<?, ?> second)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444559784
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+ Preconditions.checkArgument(
+ !FORMAT_MODELS.containsKey(key),
+ "Cannot register %s: %s is registered for format=%s model=%s",
+ formatModel.getClass(),
+ FORMAT_MODELS.containsKey(key) ? FORMAT_MODELS.get(key).getClass() : null,
+ key.first(),
+ key.second());
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D, S> WriteBuilder<D, S> writeBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, Object> factory = factoryForPositionDelete(format);
+ WriteBuilder<PositionDelete<?>, Object> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444535224
##########
core/src/main/java/org/apache/iceberg/formats/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <S> the type of the schema for the input data
+ * @param <D> the type of data records the writer will accept
+ */
+abstract class ContentFileWriteBuilderImpl<B extends ContentFileWriteBuilder<B>, D, S>
+ implements ContentFileWriteBuilder<B> {
+ private final WriteBuilder writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ static <D, S> DataWriteBuilder<D, S> forDataFile(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ return new DataFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> EqualityDeleteWriteBuilder<D, S> forEqualityDelete(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ return new EqualityDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static PositionDeleteWriteBuilder forPositionDelete(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ return new PositionDeleteFileWriteBuilder(writeBuilder, location, format);
+ }
+
+ private ContentFileWriteBuilderImpl(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public B set(String property, String value) {
+ writeBuilder.set(property, value);
+ return self();
+ }
+
+ @Override
+ public B meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return self();
+ }
+
+ @Override
+ public B metricsConfig(MetricsConfig metricsConfig) {
+ writeBuilder.metricsConfig(metricsConfig);
+ return self();
+ }
+
+ @Override
+ public B overwrite() {
+ writeBuilder.overwrite();
+ return self();
+ }
+
+ @Override
+ public B withFileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.withFileEncryptionKey(encryptionKey);
+ return self();
+ }
+
+ @Override
+ public B withAADPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.withAADPrefix(aadPrefix);
+ return self();
+ }
+
+ @Override
+ public B spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return self();
+ }
+
+ @Override
+ public B partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return self();
+ }
+
+ @Override
+ public B keyMetadata(EncryptionKeyMetadata newKeyMetadata) {
+ this.keyMetadata = newKeyMetadata;
+ return self();
+ }
+
+ @Override
+ public B sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return self();
+ }
+
+ private static class DataFileWriteBuilder<D, S>
+ extends ContentFileWriteBuilderImpl<DataWriteBuilder<D, S>, D, S>
+ implements DataWriteBuilder<D, S> {
+ private DataFileWriteBuilder(WriteBuilder writeBuilder, String location, FileFormat format) {
+ super(writeBuilder, location, format);
+ }
+
+ @Override
+ public DataFileWriteBuilder<D, S> schema(Schema schema) {
+ super.writeBuilder.schema(schema);
+ return this;
+ }
+
+ @Override
+ public DataFileWriteBuilder<D, S> inputSchema(S schema) {
+ super.writeBuilder.inputSchema(schema);
+ return this;
+ }
+
+ @Override
+ public DataFileWriteBuilder<D, S> self() {
+ return this;
+ }
+
+ @Override
+ public DataWriter<D> build() throws IOException {
+ Preconditions.checkArgument(super.spec != null, "Cannot create data writer without spec");
+ Preconditions.checkArgument(
+ super.spec.isUnpartitioned() || super.partition != null,
+ "Partition must not be null when creating data writer for partitioned spec");
+
+ return new DataWriter<>(
+ super.writeBuilder.build(),
+ super.format,
+ super.location,
+ super.spec,
+ super.partition,
+ super.keyMetadata,
+ super.sortOrder);
+ }
+ }
+
+ private static class EqualityDeleteFileWriteBuilder<D, S>
+ extends ContentFileWriteBuilderImpl<EqualityDeleteWriteBuilder<D, S>, D, S>
+ implements EqualityDeleteWriteBuilder<D, S> {
+ private Schema rowSchema = null;
+ private int[] equalityFieldIds = null;
+
+ private EqualityDeleteFileWriteBuilder(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ super(writeBuilder, location, format);
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> schema(Schema schema) {
+ super.writeBuilder.schema(schema);
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> inputSchema(S schema) {
+ super.writeBuilder.inputSchema(schema);
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> self() {
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> rowSchema(Schema schema) {
+ this.rowSchema = schema;
Review Comment:
Yeah, that's a mistake. Forgot to remove the schema method
##########
core/src/test/java/org/apache/iceberg/formats/TestFormatModelRegistry.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.AssertionsForInterfaceTypes.assertThat;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.util.Pair;
+import org.junit.jupiter.api.Test;
+
+class TestFormatModelRegistry {
+ /** Tests that registering the same class with the same configuration is successful. */
+ @Test
+ void testSuccessfulReRegister() {
+ // It is fine to register the same classes with the same configuration multiple times
+ DummyFormatModel model = new DummyFormatModel(Object.class, Object.class);
Review Comment:
Renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444537394
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder meta(String property, String value);
+
+ /**
+ * Sets multiple file metadata properties in the created file.
+ *
+ * @param properties file metadata properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /**
+ * Based on the target file content the generated {@link FileAppender} needs different
+ * configuration.
+ */
+ WriteBuilder content(FileContent content);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ WriteBuilder metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ WriteBuilder overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ WriteBuilder withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the reader
Review Comment:
Fixed
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
+ throw new UnsupportedOperationException("Not implemented yet");
+ }
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ ReadBuilder caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param filter the filter to set
+ */
+ ReadBuilder filter(Expression filter);
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ ReadBuilder set(String key, String value);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444542177
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
Review Comment:
Added comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2413098250
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
Review Comment:
IIUC then`getName` handles anonymous classes, but `getCanonicalName` returns `null` for anonymous or local classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2066040536
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
+
+ /**
+ * Registers a new object model.
+ *
+ * @param objectModel the object model
+ * @throws IllegalArgumentException if an object model for the given {@code format} and {@code
+ * objectModelName} combination already exists
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerObjectModel(ObjectModel<?> objectModel) {
+ try {
+ Key key = new Key(objectModel.format(), objectModel.name());
+ if (OBJECT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model %s clashes with %s. Both serves %s",
+ objectModel.getClass(), OBJECT_MODELS.get(key), key));
+ }
+
+ OBJECT_MODELS.put(key, objectModel);
+ } catch (RuntimeException e) {
+ // failing to register an object model is normal and does not require a stack trace
+ LOG.info(
+ "Unable to use register object model {} for data files: {}", objectModel, e.getMessage());
+ }
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register an object model is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private ObjectModelRegistry() {}
+
+ /**
+ * Provides a reader builder for the given input file which returns objects generated by the given
+ * object model.
+ *
+ * @param format of the file to read
+ * @param objectModelName name of the object model used to generate the reader
+ * @param inputFile to read
+ * @return {@link ReadBuilder} for building the actual reader
+ */
+ public static ReadBuilder<?> readBuilder(
+ FileFormat format, String objectModelName, InputFile inputFile) {
+ return OBJECT_MODELS.get(new Key(format, objectModelName)).readBuilder(inputFile);
+ }
+
+ /**
+ * Provides an appender builder for the given output file which writes a data file with a given
+ * file format and expects input records defined by the object model.
+ *
+ * @param format of the file to write
+ * @param objectModelName name of the object model used to generate the writer
+ * @param outputFile to write
+ * @param <E> type for the engine specific schema expected by the appender
+ * @return {@link ReadBuilder} for building the actual reader
+ */
+ public static <E> org.apache.iceberg.data.AppenderBuilder<?, E> appenderBuilder(
Review Comment:
Same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2067958054
##########
data/src/main/java/org/apache/iceberg/data/PositionDeleteWriterBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * Builder for generating an {@link PositionDeleteWriter}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface PositionDeleteWriterBuilder<B extends PositionDeleteWriterBuilder<B, E>, E>
+ extends FileWriterBuilderBase<B, E> {
+ /** Sets the row schema for the delete writers. */
+ B withRowSchema(Schema newSchema);
+
Review Comment:
Oh, right, this should only focus on format part.
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
+
+ /**
+ * Registers a new object model.
+ *
+ * @param objectModel the object model
+ * @throws IllegalArgumentException if an object model for the given {@code format} and {@code
+ * objectModelName} combination already exists
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerObjectModel(ObjectModel<?> objectModel) {
Review Comment:
Seems reasonable to me. Another point is should we consider throwing an exception if the `ObjectModel` has been registered? Current behavior just prints a message.
##########
data/src/main/java/org/apache/iceberg/data/AppenderBuilder.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder for generating a {@link FileAppender}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E>
Review Comment:
Oh, sorry for misclarification. I mean we already have `AppenderBuilder` in iceberg-core, why we need another one here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2078064222
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
wondering why this is not called `WriteBuilder`?
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
Review Comment:
why we are introducing a deprecated method in a brand new interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2075469793
##########
data/src/main/java/org/apache/iceberg/data/AppenderBuilder.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder for generating a {@link FileAppender}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E>
Review Comment:
I think it's better this way to limit the impact on the existing code. It's not perfect but better than refactoring everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2100237439
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If the reader does not support encryption,
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (aad) prefix used for writing the file. If the reader
+ * does not support encryption, then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the engine-specific schema for the input data records.
+ *
+ * <p>This method is necessary when the mapping between engine types and Iceberg types is not
+ * one-to-one. For example, when multiple engine types could map to the same Iceberg type, or when
+ * schema metadata beyond the structure is needed to properly interpret the data.
+ *
+ * <p>While the Iceberg schema defines the expected output structure, the engine schema provides
+ * the exact input format details needed for proper type conversion.
+ *
+ * @param newEngineSchema the native schema representation from the engine (Spark, Flink, etc.)
+ * @return this builder for method chaining
+ */
+ B dataSchema(E newEngineSchema);
+
+ /** Finalizes the configuration and builds the {@link FileAppender}. */
+ <D> FileAppender<D> build() throws IOException;
Review Comment:
The `WriteBuilder` appender will change the input type based on the ContentType parameter passed on the constructor. If `D` is the type of the object model records, then for `DATA` and `EQUALITY_DELETES` the appender will expect `D`, but for `POSITION_DELETES` it will expect `PositionDelete<D>`.
Also this would change the method signature for the existing ORC/Parquet/Avro`<D> CloseableIterable<D> ReadBuilder.build()`, <D> FileAppender<D> WriteBuilder.build()` methods. This makes creating backward compatible change even more convoluted, as we will need to create a new class, and map all the calls in the old one.
We could explore this direction as well, but this deviates even further from the current solution, and we need strong community approval for it. One possibility what I can see is to extend the `FileAccessFactory` API:
```
public interface FileAccessFactory<E, D> {
FileFormat format();
String objectModeName();
<B extends ReadBuilder<B, D>> B readBuilder(InputFile inputFile);
// For Writing D (Data and equality delete)
<B extends WriteBuilder<B, E, D>> B writeBuilder(OutputFile outputFile, FileContent content);
// For Writing PositionDelete<D>
<B extends WriteBuilder<B, E, PositionDelete<D>>> B positionDeleteWriteBuilder(OutputFile outputFile, FileContent content);
}
```
I agree that the API would be nicer/cleaner. My question is "does it worth it"? Will I have reviewers who check the changes for this level of change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2166284947
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ */
+public interface FileAccessFactory<E, D> {
Review Comment:
Reverted anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163718598
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user-provided setting if it is not supported by them.
+ */
+ default B reuseContainers() {
+ // Skip container reuse configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata and not coming from the data files themselves. The keys of the map are the column
+ * ids, the values are the accessors generating the values.
+ */
+ B constantFieldAccessors(Map<Integer, ?> constantFieldAccessors);
+
+ /** Sets a mapping from external schema names to Iceberg type IDs. */
+ B nameMapping(NameMapping newNameMapping);
+
+ /**
+ * Sets the file encryption key used for reading the file. If the reader does not support
+ * encryption, then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
Review Comment:
I followed the pattern that if something is not mandatory on the API then I provided a default implementation for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163727678
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
+ *
+ * <ul>
+ * <li><strong>generic</strong> - for {@link Record} objects (engine-agnostic)
+ * <li><strong>spark</strong> - for Apache Spark InternalRow objects
+ * <li><strong>spark-vectorized</strong> - for columnar batch processing in Spark (not supported
+ * with {@link FileFormat#AVRO})
+ * <li><strong>flink</strong> - for Apache Flink RowData objects
+ * <li><strong>arrow</strong> - for Apache Arrow columnar format (only supported with {@link
+ * FileFormat#PARQUET})
+ * </ul>
+ *
+ * <p>Processing engines can implement custom object models to integrate with Iceberg's file reading
+ * and writing capabilities.
+ *
+ * @param <E> input schema type used when converting input data to the file format
Review Comment:
For Spark:
- E is StructType
- D is InternalRow/ColumnarBatch
For Flink:
- E is RowType
- D is RowData
E supposed to be engine specific object model type
D supposed to be data type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162700119
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
Review Comment:
Set the projection schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2096242698
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
oh. this also include the `Engine Internal Object Model` dimension. instead of bundling both dimensions in one interface, I am wondering if it can be split to two separate interfaces: one for file format access factory and another for engine specific model. maybe the engine specific implementation can encapsulate the file format access factory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095464916
##########
data/src/main/java/org/apache/iceberg/data/FileWriteBuilderBase.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * Builder for generating one of the following:
+ *
+ * <ul>
+ * <li>{@link DataWriter}
+ * <li>{@link EqualityDeleteWriter}
+ * <li>{@link PositionDeleteWriter}
+ * </ul>
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+interface FileWriteBuilderBase<B extends FileWriteBuilderBase<B, E>, E>
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095261258
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * The name of the object model. Allows users to specify the object model to map the data file for
+ * reading and writing.
+ */
+ String name();
+
+ /**
+ * The appender builder for the output file which writes the data in the specified file format and
+ * accepts the records defined by this object model. The 'mode' parameter defines the input type
+ * for the specific writer use-cases. The appender should handle the following input in the
+ * specific modes:
+ *
+ * <ul>
+ * <li>The appender's engine specific input type
+ * <ul>
+ * <li>{@link WriteMode#DATA_WRITER}
+ * <li>{@link WriteMode#EQUALITY_DELETE_WRITER}
+ * </ul>
+ * <li>{@link org.apache.iceberg.deletes.PositionDelete} where the type of the row is the
+ * appender's engine specific input type when the 'mode' is {@link
+ * WriteMode#POSITION_DELETE_WRITER}
+ * </ul>
+ *
+ * @param outputFile to write to
+ * @param mode for the appender
+ * @return the appender builder
+ * @param <B> The type of the appender builder
+ */
+ <B extends AppenderBuilder<B, E>> B appenderBuilder(OutputFile outputFile, WriteMode mode);
Review Comment:
This is a very good point.
The values in the Mode kept shrinking, and now it can be replaced by `FileContent`.
Thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2093626096
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
`ObjectModel` seems too generic and doesn't really capture the responsibility of this factory class. maybe `DataIOFactory`?
`FileIO` is already taken for another context. Otherwise, `FileIOFactory` may be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095199785
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
Maybe `DataFileIOFactory`? But I would like to hear more voices on this.
I definitely don't like `FileIOFactory`. We don't create `FileIO`s with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095261258
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * The name of the object model. Allows users to specify the object model to map the data file for
+ * reading and writing.
+ */
+ String name();
+
+ /**
+ * The appender builder for the output file which writes the data in the specified file format and
+ * accepts the records defined by this object model. The 'mode' parameter defines the input type
+ * for the specific writer use-cases. The appender should handle the following input in the
+ * specific modes:
+ *
+ * <ul>
+ * <li>The appender's engine specific input type
+ * <ul>
+ * <li>{@link WriteMode#DATA_WRITER}
+ * <li>{@link WriteMode#EQUALITY_DELETE_WRITER}
+ * </ul>
+ * <li>{@link org.apache.iceberg.deletes.PositionDelete} where the type of the row is the
+ * appender's engine specific input type when the 'mode' is {@link
+ * WriteMode#POSITION_DELETE_WRITER}
+ * </ul>
+ *
+ * @param outputFile to write to
+ * @param mode for the appender
+ * @return the appender builder
+ * @param <B> The type of the appender builder
+ */
+ <B extends AppenderBuilder<B, E>> B appenderBuilder(OutputFile outputFile, WriteMode mode);
Review Comment:
This is a very good point.
The values in the Mode kept shrinking, and now they can be replaced by `FileContent`.
Thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254957038
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
Review Comment:
It looks like this exception message is based on the `InternalData` one. That specified metadata files to give additional context, but this could be for many different things so I'd probably remove the "for data files".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257229217
##########
core/src/main/java/org/apache/iceberg/data/EqualityDeleteWriteBuilder.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+
+/**
+ * A specialized builder for creating equality-based delete file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link EqualityDeleteWriter} instances.
+ *
+ * <p>The builder provides methods to configure which fields should be used for equality comparison
+ * through {@link #equalityFieldIds(List)} or {@link #equalityFieldIds(int...)}, along with schema
+ * configuration for the delete records.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the type of data records the writer will accept
+ */
+public interface EqualityDeleteWriteBuilder<B extends EqualityDeleteWriteBuilder<B, D>, D>
+ extends ContentFileWriteBuilder<B> {
+ /** Sets the row schema for the delete writers. */
+ B rowSchema(Schema rowSchema);
+
+ /** Sets the equality field ids for the equality delete writer. */
+ B equalityFieldIds(List<Integer> fieldIds);
+
+ /** Sets the equality field ids for the equality delete writer. */
+ B equalityFieldIds(int... fieldIds);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257574911
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.dataBuilder(outputFile.encryptingOutputFile());
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D> DataWriteBuilder<?, D> dataWriteBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ WriteBuilder<?, D> writeBuilder =
+ factory.equalityDeleteBuilder(outputFile.encryptingOutputFile());
Review Comment:
Thanks for catching this.
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#issuecomment-3161296766
Thanks for the review @rdblue!
My original intent was to keep the current Writer/Reader builder interfaces and implementations.
Based on your comments, I feel we might want to consider creating completely new ReadBuilder/WriterBuilder object instead of keeping the old ones as much as possible. Am I right? That would be an even bigger change on the implementation side, as we need to create the new builders and either duplicate the current reader/writer code, or make sure that the implementation share as much as possible. Are we ready for such a big change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2463936868
##########
core/src/main/java/org/apache/iceberg/formats/DataWriteBuilder.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A specialized builder for creating data content file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link DataWriter} instances. Data writers produce table content files
+ * containing actual data records stored in an Iceberg table, configured according to the table's
+ * schema and partition specification.
+ *
+ * @param <D> the type of data records the writer will accept
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2726933425
##########
core/src/main/java/org/apache/iceberg/formats/CommonWriteBuilder.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface CommonWriteBuilder<B extends CommonWriteBuilder<B>> {
Review Comment:
1 minor release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775155734
##########
core/src/main/java/org/apache/iceberg/formats/BaseFormatModel.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.Map;
+import org.apache.iceberg.Schema;
+
+/**
+ * Base implementation of {@link FormatModel} that provides common functionality for format models.
+ *
+ * <p>This abstract class serves as a foundation for creating format-specific models that handle
+ * reading and writing data in various file formats.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ * @param <W> the writer type produced by the writer function
+ * @param <R> the reader type produced by the reader function
+ * @param <F> the file schema type used by the underlying file format
+ */
+public abstract class BaseFormatModel<D, S, W, R, F> implements FormatModel<D, S> {
+ private final Class<? extends D> type;
+ private final Class<S> schemaType;
+ private final WriterFunction<W, S, F> writerFunction;
+ private final ReaderFunction<R, S, F> readerFunction;
+
+ /**
+ * Constructs a new BaseFormatModel with the specified configuration.
+ *
+ * @param type the row type class for the object model implementation processed by this factory.
+ * @param schemaType the schema type class for the object model implementation processed by this
+ * factory.
+ * @param writerFunction the function used to create writers for this format
+ * @param readerFunction the function used to create readers for this format
+ */
+ protected BaseFormatModel(
+ Class<? extends D> type,
+ Class<S> schemaType,
+ WriterFunction<W, S, F> writerFunction,
+ ReaderFunction<R, S, F> readerFunction) {
+ this.type = type;
+ this.schemaType = schemaType;
+ this.writerFunction = writerFunction;
+ this.readerFunction = readerFunction;
+ }
+
+ @Override
+ public Class<? extends D> type() {
+ return type;
+ }
+
+ @Override
+ public Class<S> schemaType() {
+ return schemaType;
+ }
+
+ /**
+ * Returns the writer function used to create writers for this format.
+ *
+ * @return the writer function
+ */
+ protected WriterFunction<W, S, F> writerFunction() {
+ return writerFunction;
+ }
+
+ /**
+ * Returns the reader function used to create readers for this format.
+ *
+ * @return the reader function
+ */
+ protected ReaderFunction<R, S, F> readerFunction() {
+ return readerFunction;
+ }
+
+ /**
+ * A functional interface for creating writers that can write data in a specific format.
+ *
+ * @param <W> the writer type to be created
+ * @param <S> the type of the schema for the input data
Review Comment:
Engine-specific schema describing the input records (optional).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2776218568
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and {@link
+ * FileWriterBuilder}s for writing various types of Iceberg content files. The appropriate builder
+ * is selected based on {@link FileFormat} and object model name.
Review Comment:
Nit: object model class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2768968383
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and {@link
+ * FileWriterBuilder}s for writing various types of Iceberg content files. The appropriate builder
+ * is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private FormatModelRegistry() {}
+
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ // Format models indexed by file format and object model class
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static synchronized void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null,
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @param <S> the type of the output schema for the reader
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<? extends D> type, InputFile inputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return model.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> FileWriterBuilder<DataWriter<D>, S> dataWriteBuilder(
+ FileFormat format, Class<? extends D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return FileWriterBuilderImpl.forDataFile(model, outputFile);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> FileWriterBuilder<EqualityDeleteWriter<D>, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> model = modelFor(format, type);
+ return FileWriterBuilderImpl.forEqualityDelete(model, outputFile);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public static <D> FileWriterBuilder<PositionDeleteWriter<D>, ?> positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<D>, ?> model =
+ (FormatModel<PositionDelete<D>, ?>) (FormatModel) modelFor(format, PositionDelete.class);
Review Comment:
This ugliness remains. IntelliJ does not report an error if the cast is removed, but the code fails to compile:
```
> Task :iceberg-core:compileJava
/Users/petervary/dev/iceberg/core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:182: error: incompatible types: cannot infer type-variable(s) D#1,S
FormatModel<PositionDelete<D>, ?> model = modelFor(format, PositionDelete.class);
^
(argument mismatch; Class<PositionDelete> cannot be converted to Class<? extends PositionDelete<D#2>>)
where D#1,S,D#2 are type-variables:
D#1 extends Object declared in method <D#1,S>modelFor(FileFormat,Class<? extends D#1>)
S extends Object declared in method <D#1,S>modelFor(FileFormat,Class<? extends D#1>)
D#2 extends Object declared in method <D#2>positionDeleteWriteBuilder(FileFormat,EncryptedOutputFile)
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 error
```
The available alternatives are:
- Casting the input twice:
```
FormatModel<PositionDelete<D>, ?> model =
modelFor(format, (Class<PositionDelete<D>>) (Class) PositionDelete.class);
```
- Bypassing using the `modelFor` method:
```
FormatModel<PositionDelete<D>, ?> model =
(FormatModel<PositionDelete<D>, ?>) MODELS.get(Pair.of(format, PositionDelete.class));
Preconditions.checkArgument(
model != null,
"Format model is not registered for format %s and type %s",
format,
PositionDelete.class);
```
Any ideas, suggestions welcome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2350292523
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <S> the type of the schema for the input data
+ * @param <D> the type of data records the writer will accept
+ */
+@SuppressWarnings("unchecked")
+abstract class ContentFileWriteBuilderImpl<B extends ContentFileWriteBuilder<B, S>, D, S>
+ implements ContentFileWriteBuilder<B, S> {
+ private final WriteBuilder<D, S> writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ static <D, S> DataWriteBuilder<D, S> forDataFile(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new DataFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> EqualityDeleteWriteBuilder<D, S> forEqualityDelete(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new EqualityDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> PositionDeleteWriteBuilder<D, S> forPositionDelete(
+ WriteBuilder<PositionDelete<D>, S> writeBuilder, String location, FileFormat format) {
+ return new PositionDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ private ContentFileWriteBuilderImpl(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public B set(String property, String value) {
+ writeBuilder.set(property, value);
+ return self();
+ }
+
+ @Override
+ public B meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return self();
+ }
+
+ @Override
+ public B metricsConfig(MetricsConfig metricsConfig) {
+ writeBuilder.metricsConfig(metricsConfig);
+ return self();
+ }
+
+ @Override
+ public B overwrite() {
+ writeBuilder.overwrite();
+ return self();
+ }
+
+ @Override
+ public B fileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.fileEncryptionKey(encryptionKey);
+ return self();
+ }
+
+ @Override
+ public B fileAADPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.fileAADPrefix(aadPrefix);
+ return self();
+ }
+
+ @Override
+ public B spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return self();
+ }
+
+ @Override
+ public B partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return self();
+ }
+
+ @Override
+ public B keyMetadata(EncryptionKeyMetadata newKeyMetadata) {
+ this.keyMetadata = newKeyMetadata;
+ return self();
+ }
+
+ @Override
+ public B sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return self();
+ }
+
+ private static class DataFileWriteBuilder<D, S>
Review Comment:
Both `ContentFileWriteBuilderImpl` and `DataWriteBuilder` extends from or implements `ContentFileWriteBuilderImpl` interface. wondering if it is a sign that if the hierarchy can be improved.
E.g., one solution is to have `DataWriteBuilder` extends from the abstract class `ContentFileWriteBuilderImpl`. the down side is that `DataWriteBuilder` couldn't be an interface anymore, which is also non ideal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2227774387
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ B project(Schema newSchema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param newCaseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean newCaseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ B filter(Expression newFilter);
Review Comment:
would be good to also omit the `new` prefix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228095255
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
+
+ /** Set the file schema. */
+ B fileSchema(Schema newSchema);
Review Comment:
Moved to `fileSchema` so it is consistent everywhere on the API. The parameter name is the same as the method name
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
+
+ /** Set the file schema. */
+ B fileSchema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ B set(Map<String, String> properties);
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /**
+ * Add the new properties to file metadata for the created file.
+ *
+ * @param properties a map of file metadata properties
+ * @return this for method chaining
+ */
+ B meta(Map<String, String> properties);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2372937189
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B, S>, S> {
Review Comment:
currently `data` sub package in the core module only contains Iceberg GenericRecord related and avro. `io` sub packages contain more related classes like `PartitionedWriter`, `FanoutWriter` etc. It would be great to have others weigh in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2546948615
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -229,4 +206,27 @@ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> t
model != null, "Format model is not registered for format %s and type %s", format, type);
return model;
}
+
+ private static boolean equals(FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
Review Comment:
nit: I know this is INFO level. The only reason this can fail is that the class can't be found/loaded. maybe the error msg can be sth like.
```
LOG.INFO("Skip registration of {}. Likely the jar is not in the classpath", classToRegister);
```
`Unable ...` and the stack trace would sound alarming to users (although INFO level). And this happens 100% for at least one log line for either Spark or Flink format model (not found) in any engine/client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2445664919
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
Review Comment:
I am favoring removing it for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2440383763
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, ?> factory = factoryForPositionDelete(format);
+ WriteBuilder writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @VisibleForTesting
+ static Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> models() {
+ return MODELS;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
+ FormatModel<D, S> model = ((FormatModel<D, S>) MODELS.get(Pair.of(format, type)));
+ Preconditions.checkNotNull(model, "Format model is not registered for %s and %s", format, type);
Review Comment:
Moved to `Preconditions.checkArgument`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2449364546
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
Review Comment:
this is the drawback of not having generic parameters.
@rdblue Should we revisit @pvary 's [earlier attempt](https://github.com/pvary/iceberg/commit/8f9a76afca353547b3e66f865948562115d78546#diff-80cf1d6f419745a3356f34943e334f426bac80091b86e231e839625aa869a613) to define the new interfaces with generic types.
To avoid breaking the source compatibility and keep the change scope small, the existing implementations (Avro, Orc, Parquet) would use `WriteBuilder<Object, Object>`, which means the generic parameters don't bring any materialized benefits. but at least the new interfaces are properly types. Who knows. In the future, the implementations may be updated. Also if we add a new file format support in the future, the new implementation can be properly typed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2449364546
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
Review Comment:
this is the drawback of not having generic parameters.
@rdblue Should we revisit @pvary 's [earlier attempt](https://github.com/pvary/iceberg/commit/8f9a76afca353547b3e66f865948562115d78546#diff-80cf1d6f419745a3356f34943e334f426bac80091b86e231e839625aa869a613) to define the new interfaces with generic types.
To avoid breaking the source compatibility and keep the change scope small, the existing implementations (Avro, Orc, Parquet) would use `WriteBuilder<Object, Object>`, which means the generic parameters don't bring any materialized benefits.
but at least the new interfaces are properly typed. Who knows. In the future, the implementations may be updated. Also if we add a new file format support in the future, the new implementation can be properly typed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2045976962
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
+
+ /**
+ * Registers a new object model.
+ *
+ * @param objectModel the object model
+ * @throws IllegalArgumentException if an object model for the given {@code format} and {@code
+ * objectModelName} combination already exists
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerObjectModel(ObjectModel<?> objectModel) {
Review Comment:
This implementation seems odd to me, it throws an exception to avoid duplicated registeration. How about just like this:
```
val ret = OBJECT_MODELS.putIfAbsent(k, v);
if (ret != null) {
log.info(...)
}
```
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
Review Comment:
```suggestion
private static final ConcurrentMap<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
```
If we want to allow registering non predefined formats at runtime, it should be a concurrrent map?
##########
data/src/main/java/org/apache/iceberg/data/PositionDeleteWriterBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * Builder for generating an {@link PositionDeleteWriter}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface PositionDeleteWriterBuilder<B extends PositionDeleteWriterBuilder<B, E>, E>
+ extends FileWriterBuilderBase<B, E> {
+ /** Sets the row schema for the delete writers. */
+ B withRowSchema(Schema newSchema);
+
Review Comment:
Position deletes's input maybe sorted, should we also have a config key for the case?
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
+
+ /**
+ * Registers a new object model.
+ *
+ * @param objectModel the object model
+ * @throws IllegalArgumentException if an object model for the given {@code format} and {@code
+ * objectModelName} combination already exists
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerObjectModel(ObjectModel<?> objectModel) {
+ try {
+ Key key = new Key(objectModel.format(), objectModel.name());
+ if (OBJECT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model %s clashes with %s. Both serves %s",
+ objectModel.getClass(), OBJECT_MODELS.get(key), key));
+ }
+
+ OBJECT_MODELS.put(key, objectModel);
+ } catch (RuntimeException e) {
+ // failing to register an object model is normal and does not require a stack trace
+ LOG.info(
+ "Unable to use register object model {} for data files: {}", objectModel, e.getMessage());
Review Comment:
```suggestion
"Unable to register object model {} for data files: {}", objectModel, e.getMessage());
```
##########
data/src/main/java/org/apache/iceberg/data/AppenderBuilder.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder for generating a {@link FileAppender}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E>
Review Comment:
Do we really need this?
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
+
+ /**
+ * Registers a new object model.
+ *
+ * @param objectModel the object model
+ * @throws IllegalArgumentException if an object model for the given {@code format} and {@code
+ * objectModelName} combination already exists
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerObjectModel(ObjectModel<?> objectModel) {
+ try {
+ Key key = new Key(objectModel.format(), objectModel.name());
+ if (OBJECT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model %s clashes with %s. Both serves %s",
+ objectModel.getClass(), OBJECT_MODELS.get(key), key));
+ }
+
+ OBJECT_MODELS.put(key, objectModel);
+ } catch (RuntimeException e) {
+ // failing to register an object model is normal and does not require a stack trace
+ LOG.info(
+ "Unable to use register object model {} for data files: {}", objectModel, e.getMessage());
+ }
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register an object model is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private ObjectModelRegistry() {}
+
+ /**
+ * Provides a reader builder for the given input file which returns objects generated by the given
+ * object model.
+ *
+ * @param format of the file to read
+ * @param objectModelName name of the object model used to generate the reader
+ * @param inputFile to read
+ * @return {@link ReadBuilder} for building the actual reader
+ */
+ public static ReadBuilder<?> readBuilder(
+ FileFormat format, String objectModelName, InputFile inputFile) {
+ return OBJECT_MODELS.get(new Key(format, objectModelName)).readBuilder(inputFile);
+ }
+
+ /**
+ * Provides an appender builder for the given output file which writes a data file with a given
+ * file format and expects input records defined by the object model.
+ *
+ * @param format of the file to write
+ * @param objectModelName name of the object model used to generate the writer
+ * @param outputFile to write
+ * @param <E> type for the engine specific schema expected by the appender
+ * @return {@link ReadBuilder} for building the actual reader
+ */
+ public static <E> org.apache.iceberg.data.AppenderBuilder<?, E> appenderBuilder(
Review Comment:
Do we really need this?
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which should be implemented by the data file format implementations. The {@link
+ * AppenderBuilder} will be parametrized based on the user provided configuration and finally the
+ * {@link AppenderBuilder#build(AppenderBuilder.WriteMode)} method is used to generate the appender
+ * for the specific writer use-cases. The following input should be handled by the appender in the
+ * specific modes:
+ *
+ * <ul>
+ * <li>The appender's engine specific input type
+ * <ul>
+ * <li>{@link AppenderBuilder.WriteMode#APPENDER}
+ * <li>{@link AppenderBuilder.WriteMode#DATA_WRITER}
+ * <li>{@link AppenderBuilder.WriteMode#EQUALITY_DELETE_WRITER}
+ * </ul>
+ * <li>{@link org.apache.iceberg.deletes.PositionDelete} where the type of the row is the
+ * appender's engine specific input type
+ * <ul>
+ * <li>{@link AppenderBuilder.WriteMode#POSITION_DELETE_WRITER}
+ * <li>{@link AppenderBuilder.WriteMode#POSITION_DELETE_WITH_ROW_WRITER}
+ * </ul>
+ * </ul>
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If encryption is not supported by the reader
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data prefix used for writing the file. If encryption is not
+ * supported by the reader then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the engine native schema for the input. Defines the input type when there is N to 1
+ * mapping between the engine type and the Iceberg type, and providing the Iceberg schema is not
+ * enough for the conversion.
+ */
+ B engineSchema(E newEngineSchema);
+
+ /**
+ * Builds the {@link FileAppender} for the configured {@link WriteMode}. Could change several
+ * use-case specific configurations, like:
+ *
+ * <ul>
+ * <li>Mode specific writer context (typically different for data and delete files).
+ * <li>Writer functions to accept data rows, or {@link
+ * org.apache.iceberg.deletes.PositionDelete}s
+ * </ul>
+ */
+ <D> FileAppender<D> build(WriteMode mode) throws IOException;
+
+ /**
+ * Writer modes. Based on the mode {@link #build(WriteMode)} could alter the appender
+ * configuration when creating the {@link FileAppender}.
+ */
+ enum WriteMode {
+ /** Mode for appending data to a file. */
+ APPENDER,
Review Comment:
How is this different from `DATA_WRITER`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2065986466
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
+ * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
+ * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
+ * and the requested `object model name` the registry returns the correct reader and writer
+ * builders. These builders could be used to generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones are wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Key, ObjectModel<?>> OBJECT_MODELS = Maps.newConcurrentMap();
+
+ /**
+ * Registers a new object model.
+ *
+ * @param objectModel the object model
+ * @throws IllegalArgumentException if an object model for the given {@code format} and {@code
+ * objectModelName} combination already exists
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerObjectModel(ObjectModel<?> objectModel) {
Review Comment:
@danielcweeks and @rdblue specifically asked for harder guardrails around registering object models. This check prevents users to accidentally overwrite embedded object models.
If the community has consensus around changing this behavior, I'm happy to do either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2077289136
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -36,10 +36,10 @@
/**
* Registry which provides the available {@link ReadBuilder}s and writer builders ({@link
- * org.apache.iceberg.data.AppenderBuilder}, {@link DataWriterBuilder}, {@link
- * EqualityDeleteWriterBuilder}, {@link PositionDeleteWriterBuilder}). Based on the `file format`
- * and the requested `object model name` the registry returns the correct reader and writer
- * builders. These builders could be used to generate the readers and writers.
+ * AppenderBuilder}, {@link DataWriterBuilder}, {@link EqualityDeleteWriterBuilder}, {@link
Review Comment:
```suggestion
{@link DataWriterBuilder}, {@link EqualityDeleteWriterBuilder}, {@link
```
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -49,9 +49,13 @@
public final class ObjectModelRegistry {
private static final Logger LOG = LoggerFactory.getLogger(ObjectModelRegistry.class);
// The list of classes which are used for registering the reader and writer builders
- private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+ private static final List<String> CLASSES_TO_REGISTER =
+ ImmutableList.of(
+ "org.apache.iceberg.arrow.vectorized.ArrowReader",
+ "org.apache.iceberg.flink.data.FlinkObjectModels",
+ "org.apache.iceberg.spark.source.SparkObjectModels");
Review Comment:
I would prefer to add them one by one when we implement each model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2098385095
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
sounds good, this seems like a good choice for now. `FileAppender` and `FileWriter` renaming are disruptive and probably can be discussed separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2100279569
##########
data/src/main/java/org/apache/iceberg/data/PositionDeleteWriteBuilder.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * A specialized builder for creating position-based delete file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link PositionDeleteWriter} instances.
+ *
+ * <p>The builder provides methods to configure the schema for the row data that might be included
+ * with the position deletes through {@link #rowSchema(Schema)}, enabling optional preservation of
+ * deleted record content.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type required by the writer for data conversion
+ */
+public interface PositionDeleteWriteBuilder<B extends PositionDeleteWriteBuilder<B, E>, E>
+ extends ContentFileWriteBuilder<B, E> {
+ /** Sets the row schema for the delete writers. */
+ B rowSchema(Schema newSchema);
Review Comment:
Also moved/rewritten the javadoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2099750595
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link ReadBuilder} instances based on:
+ *
+ * <ul>
+ * <li>source file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * </ul>
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+public interface ReadBuilder<B extends ReadBuilder<B>> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
Review Comment:
I think the current api is misleading. The case sensitivity is not working with column names. The columns are identified by ids, or name mapping, but never by names. So the reader only uses the case sensitivity for the filters. I wanted to highlight this with this change in the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2099762112
##########
data/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.WriteBuilder;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type required by the writer for data conversion
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B, E>, E> {
Review Comment:
I intentionally try to keep a very separate API for the content file writers from the actual writers. In my mental model they are just "accidentally" similar. One is for the file formats to implement, one is for the users to write content files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095316334
##########
data/src/main/java/org/apache/iceberg/data/FileWriteBuilderBase.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * Builder for generating one of the following:
+ *
+ * <ul>
+ * <li>{@link DataWriter}
+ * <li>{@link EqualityDeleteWriter}
+ * <li>{@link PositionDeleteWriter}
+ * </ul>
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+interface FileWriteBuilderBase<B extends FileWriteBuilderBase<B, E>, E>
Review Comment:
Good point.
After removing the Appender from the data module, we don't need the base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2097739913
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
Change the `AppenderBuilder` to `WriteBuilder`, but kept the `FileAppender` as it is.
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
Changed the `AppenderBuilder` to `WriteBuilder`, but kept the `FileAppender` as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2096242698
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
oh. this also include the `Engine Internal Object Model` dimension. instead of bundling both dimensions in one interface, I am wondering if it can be split to two separate interfaces: one for file format access factory and another for engine specific model. maybe the engine specific implementation can encapsulate the file format access factory as a class member.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2441157815
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ WriteBuilder meta(String property, String value);
+
+ /**
+ * Sets multiple file metadata properties in the created file.
+ *
+ * @param properties file metadata properties to set
+ * @return this for method chaining
+ */
+ default WriteBuilder meta(Map<String, String> properties) {
+ properties.forEach(this::meta);
+ return this;
+ }
+
+ /**
+ * Based on the target file content the generated {@link FileAppender} needs different
+ * configuration.
+ */
+ WriteBuilder content(FileContent content);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ WriteBuilder metricsConfig(MetricsConfig metricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ WriteBuilder overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ WriteBuilder withFileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the reader
+ * does not support encryption, then an exception should be thrown.
+ */
+ WriteBuilder withAADPrefix(ByteBuffer aadPrefix);
+
+ /** Finalizes the configuration and builds the {@link FileAppender}. */
+ <D> FileAppender<D> build() throws IOException;
Review Comment:
do we still need this with the FileAppenderFactory deprecation merged?
##########
core/src/test/java/org/apache/iceberg/formats/TestFormatModelRegistry.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.AssertionsForInterfaceTypes.assertThat;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.util.Pair;
+import org.junit.jupiter.api.Test;
+
+class TestFormatModelRegistry {
+ /** Tests that registering the same class with the same configuration is successful. */
+ @Test
+ void testSuccessfulReRegister() {
+ // It is fine to register the same classes with the same configuration multiple times
Review Comment:
nit: move this comment line to just before line 39
##########
core/src/main/java/org/apache/iceberg/formats/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <S> the type of the schema for the input data
+ * @param <D> the type of data records the writer will accept
+ */
+abstract class ContentFileWriteBuilderImpl<B extends ContentFileWriteBuilder<B>, D, S>
+ implements ContentFileWriteBuilder<B> {
+ private final WriteBuilder writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ static <D, S> DataWriteBuilder<D, S> forDataFile(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ return new DataFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> EqualityDeleteWriteBuilder<D, S> forEqualityDelete(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ return new EqualityDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static PositionDeleteWriteBuilder forPositionDelete(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ return new PositionDeleteFileWriteBuilder(writeBuilder, location, format);
+ }
+
+ private ContentFileWriteBuilderImpl(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public B set(String property, String value) {
+ writeBuilder.set(property, value);
+ return self();
+ }
+
+ @Override
+ public B meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return self();
+ }
+
+ @Override
+ public B metricsConfig(MetricsConfig metricsConfig) {
+ writeBuilder.metricsConfig(metricsConfig);
+ return self();
+ }
+
+ @Override
+ public B overwrite() {
+ writeBuilder.overwrite();
+ return self();
+ }
+
+ @Override
+ public B withFileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.withFileEncryptionKey(encryptionKey);
+ return self();
+ }
+
+ @Override
+ public B withAADPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.withAADPrefix(aadPrefix);
+ return self();
+ }
+
+ @Override
+ public B spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return self();
+ }
+
+ @Override
+ public B partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return self();
+ }
+
+ @Override
+ public B keyMetadata(EncryptionKeyMetadata newKeyMetadata) {
+ this.keyMetadata = newKeyMetadata;
+ return self();
+ }
+
+ @Override
+ public B sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return self();
+ }
+
+ private static class DataFileWriteBuilder<D, S>
+ extends ContentFileWriteBuilderImpl<DataWriteBuilder<D, S>, D, S>
+ implements DataWriteBuilder<D, S> {
+ private DataFileWriteBuilder(WriteBuilder writeBuilder, String location, FileFormat format) {
+ super(writeBuilder, location, format);
+ }
+
+ @Override
+ public DataFileWriteBuilder<D, S> schema(Schema schema) {
+ super.writeBuilder.schema(schema);
+ return this;
+ }
+
+ @Override
+ public DataFileWriteBuilder<D, S> inputSchema(S schema) {
+ super.writeBuilder.inputSchema(schema);
+ return this;
+ }
+
+ @Override
+ public DataFileWriteBuilder<D, S> self() {
+ return this;
+ }
+
+ @Override
+ public DataWriter<D> build() throws IOException {
+ Preconditions.checkArgument(super.spec != null, "Cannot create data writer without spec");
+ Preconditions.checkArgument(
+ super.spec.isUnpartitioned() || super.partition != null,
+ "Partition must not be null when creating data writer for partitioned spec");
+
+ return new DataWriter<>(
+ super.writeBuilder.build(),
+ super.format,
+ super.location,
+ super.spec,
+ super.partition,
+ super.keyMetadata,
+ super.sortOrder);
+ }
+ }
+
+ private static class EqualityDeleteFileWriteBuilder<D, S>
+ extends ContentFileWriteBuilderImpl<EqualityDeleteWriteBuilder<D, S>, D, S>
+ implements EqualityDeleteWriteBuilder<D, S> {
+ private Schema rowSchema = null;
+ private int[] equalityFieldIds = null;
+
+ private EqualityDeleteFileWriteBuilder(
+ WriteBuilder writeBuilder, String location, FileFormat format) {
+ super(writeBuilder, location, format);
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> schema(Schema schema) {
+ super.writeBuilder.schema(schema);
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> inputSchema(S schema) {
+ super.writeBuilder.inputSchema(schema);
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> self() {
+ return this;
+ }
+
+ @Override
+ public EqualityDeleteFileWriteBuilder<D, S> rowSchema(Schema schema) {
+ this.rowSchema = schema;
Review Comment:
`rowSchema` is eventually set to builder as `super.writeBuilder.schema(schema)` in `build()` method.
but the `schema(Schema schmea` in line 208 also does the same thing. do we need two methods for the same thing?
##########
core/src/test/java/org/apache/iceberg/formats/TestFormatModelRegistry.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.AssertionsForInterfaceTypes.assertThat;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.util.Pair;
+import org.junit.jupiter.api.Test;
+
+class TestFormatModelRegistry {
+ /** Tests that registering the same class with the same configuration is successful. */
+ @Test
+ void testSuccessfulReRegister() {
+ // It is fine to register the same classes with the same configuration multiple times
+ DummyFormatModel model = new DummyFormatModel(Object.class, Object.class);
+ FormatModelRegistry.register(model);
+ assertThat(FormatModelRegistry.models())
+ .containsEntry(Pair.of(FileFormat.PARQUET, Object.class), model);
+ FormatModelRegistry.register(model);
+ assertThat(FormatModelRegistry.models())
+ .containsEntry(Pair.of(FileFormat.PARQUET, Object.class), model);
+ }
+
+ /** Tests that registering the same class with the same configuration updates the registration. */
+ @Test
+ void testUpdatedRegistration() {
+ DummyFormatModel model1 = new DummyFormatModel(Object.class, Object.class);
+ DummyFormatModel model2 = new DummyFormatModel(Object.class, Object.class);
+ FormatModelRegistry.register(model1);
+ assertThat(FormatModelRegistry.models().get(Pair.of(model1.format(), model1.type())))
+ .isEqualTo(model1);
+
+ // Registering a new model with the same format and schema type should replace the old one
+ FormatModelRegistry.register(model2);
+ assertThat(FormatModelRegistry.models().get(Pair.of(model1.format(), model1.type())))
+ .isNotEqualTo(model1);
+ assertThat(FormatModelRegistry.models().get(Pair.of(model1.format(), model1.type())))
+ .isEqualTo(model2);
Review Comment:
#`isNotSameAs` and `isSameAs` might be more accurate for identity comparison here, although `isEqualTo` works the same here because `DummyFormatModel` doesn't override `equalsTo` method
##########
core/src/test/java/org/apache/iceberg/formats/TestFormatModelRegistry.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.AssertionsForInterfaceTypes.assertThat;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.util.Pair;
+import org.junit.jupiter.api.Test;
+
+class TestFormatModelRegistry {
+ /** Tests that registering the same class with the same configuration is successful. */
+ @Test
+ void testSuccessfulReRegister() {
+ // It is fine to register the same classes with the same configuration multiple times
+ DummyFormatModel model = new DummyFormatModel(Object.class, Object.class);
Review Comment:
nit: rename `DummyFormatModel` to `DummyParquetFormatModel`
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
Review Comment:
this doesn't seem to be used in the PoC uber PR #12298
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+ Preconditions.checkArgument(
+ !FORMAT_MODELS.containsKey(key),
+ "Cannot register %s: %s is registered for format=%s model=%s",
+ formatModel.getClass(),
+ FORMAT_MODELS.containsKey(key) ? FORMAT_MODELS.get(key).getClass() : null,
+ key.first(),
+ key.second());
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder<D, S> readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D, S> WriteBuilder<D, S> writeBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder<D, S> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, Object> factory = factoryForPositionDelete(format);
+ WriteBuilder<PositionDelete<?>, Object> writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
Review Comment:
nit: `Format model is not registered for format %s and type %s`
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
Review Comment:
nit: `the unique identifier` seems unclear to me. Maybe?
```
Return the row type class
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2440850166
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
+ throw new UnsupportedOperationException("Not implemented yet");
+ }
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ ReadBuilder caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param filter the filter to set
+ */
+ ReadBuilder filter(Expression filter);
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ ReadBuilder set(String key, String value);
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default ReadBuilder setAll(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return this;
+ }
+
+ /** Enables reusing the containers returned by the reader. Decreases pressure on GC. */
+ ReadBuilder reuseContainers();
+
+ /** Sets the batch size for vectorized readers. */
+ ReadBuilder recordsPerBatch(int numRowsPerBatch);
Review Comment:
The goal was to keep the old function names/signatures whenever it is possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444533102
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444455982
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
Review Comment:
Yes. We don't plan to change this here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2353692309
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <S> the type of the schema for the input data
+ * @param <D> the type of data records the writer will accept
+ */
+@SuppressWarnings("unchecked")
+abstract class ContentFileWriteBuilderImpl<B extends ContentFileWriteBuilder<B, S>, D, S>
+ implements ContentFileWriteBuilder<B, S> {
+ private final WriteBuilder<D, S> writeBuilder;
+ private final String location;
+ private final FileFormat format;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ static <D, S> DataWriteBuilder<D, S> forDataFile(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new DataFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> EqualityDeleteWriteBuilder<D, S> forEqualityDelete(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ return new EqualityDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ static <D, S> PositionDeleteWriteBuilder<D, S> forPositionDelete(
+ WriteBuilder<PositionDelete<D>, S> writeBuilder, String location, FileFormat format) {
+ return new PositionDeleteFileWriteBuilder<>(writeBuilder, location, format);
+ }
+
+ private ContentFileWriteBuilderImpl(
+ WriteBuilder<D, S> writeBuilder, String location, FileFormat format) {
+ this.writeBuilder = writeBuilder;
+ this.location = location;
+ this.format = format;
+ }
+
+ @Override
+ public B set(String property, String value) {
+ writeBuilder.set(property, value);
+ return self();
+ }
+
+ @Override
+ public B meta(String property, String value) {
+ writeBuilder.meta(property, value);
+ return self();
+ }
+
+ @Override
+ public B metricsConfig(MetricsConfig metricsConfig) {
+ writeBuilder.metricsConfig(metricsConfig);
+ return self();
+ }
+
+ @Override
+ public B overwrite() {
+ writeBuilder.overwrite();
+ return self();
+ }
+
+ @Override
+ public B fileEncryptionKey(ByteBuffer encryptionKey) {
+ writeBuilder.fileEncryptionKey(encryptionKey);
+ return self();
+ }
+
+ @Override
+ public B fileAADPrefix(ByteBuffer aadPrefix) {
+ writeBuilder.fileAADPrefix(aadPrefix);
+ return self();
+ }
+
+ @Override
+ public B spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return self();
+ }
+
+ @Override
+ public B partition(StructLike newPartition) {
+ this.partition = newPartition;
+ return self();
+ }
+
+ @Override
+ public B keyMetadata(EncryptionKeyMetadata newKeyMetadata) {
+ this.keyMetadata = newKeyMetadata;
+ return self();
+ }
+
+ @Override
+ public B sortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return self();
+ }
+
+ private static class DataFileWriteBuilder<D, S>
Review Comment:
`DataWriteBuilder`, `EqualityDeleteWriteBuilder` and `PositionDeleteWriteBuilder` are intended as public interfaces which we expose. The `ContentFileWriteBuilder` is only there for the common parts, so we don't have duplicate codes.
`ContentFileWriteBuilderImpl` implements the `ContentFileWriteBuilder` part of the APIs, but not intended as public. The other classes implement the specfic parts, but only exposed through the interfaces, and not as classes themselves.
Does this makes sense?
I would want to keep the `DataWriteBuilder`, `EqualityDeleteWriteBuilder` and `PositionDeleteWriteBuilder`. We might remove the `ContentFileWriteBuilder` interface but that only adds back duplicated code, so I wouldn't do that.
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254941620
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
Review Comment:
Shouldn't this be `<D, B extends WriteBuilder<B, D>> WriteBuilder<B, D> writeBuilder(...)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254958537
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.dataBuilder(outputFile.encryptingOutputFile());
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D> DataWriteBuilder<?, D> dataWriteBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ WriteBuilder<?, D> writeBuilder =
+ factory.equalityDeleteBuilder(outputFile.encryptingOutputFile());
Review Comment:
Looks like a bug. This is creating an equality delete builder instead of a data file builder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254953235
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the input data type for the writer
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, D>, D> {
+ /** Set the file schema. */
+ B fileSchema(Schema fileSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ @SuppressWarnings("unchecked")
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
Review Comment:
Using `self()` or removing the builder type would avoid the unchecked cast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257573820
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257578694
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the input data type for the writer
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, D>, D> {
Review Comment:
So the old APIs can call the old methods.
This is what I have mentioned here: https://github.com/apache/iceberg/pull/12774#discussion_r2257555813
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
Review Comment:
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2258025474
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
Review Comment:
Reverted back to have a specific method for setting the batch size:
```
B recordsPerBatch(int numRowsPerBatch);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2166950611
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user-provided setting if it is not supported by them.
+ */
+ default B reuseContainers() {
+ // Skip container reuse configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata and not coming from the data files themselves. The keys of the map are the column
+ * ids, the values are the accessors generating the values.
+ */
+ B constantFieldAccessors(Map<Integer, ?> constantFieldAccessors);
+
+ /** Sets a mapping from external schema names to Iceberg type IDs. */
+ B nameMapping(NameMapping newNameMapping);
+
+ /**
+ * Sets the file encryption key used for reading the file. If the reader does not support
+ * encryption, then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data (AAD) prefix for decryption. If the reader does not
+ * support decryption, then an exception should be thrown.
+ */
+ default B fileAADPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /** Builds the reader. */
+ CloseableIterable<D> build();
Review Comment:
I think the comment on #12476 is to handle the metadata retrieval later, and I think later is now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162701774
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user-provided setting if it is not supported by them.
+ */
+ default B reuseContainers() {
Review Comment:
I think implementations should need to consider this, rather than automatically skipping it in the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162569873
##########
build.gradle:
##########
@@ -373,6 +373,7 @@ project(':iceberg-data') {
dependencies {
implementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
api project(':iceberg-api')
+ implementation project(':iceberg-common')
Review Comment:
What is being accessed from `iceberg-common`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162622583
##########
core/src/main/java/org/apache/iceberg/io/FileAccessFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>FileAccessFactory serves as a bridge between storage formats ({@link FileFormat}) and expected
+ * input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides these built-in object models:
Review Comment:
This list shouldn't be in core, even in a comment. It will just become stale and inaccurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2163722643
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
Review Comment:
Left it here, to show how vectorized reading will be implemented. This could be a common configuration which could be used instead of the original `recordsPerBatch`. The user could use`set(String, String)` to set this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2500323241
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.InputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Return the row type class for the object model implementation processed by this factory.
+ *
+ * <p>The model types act as a contract specifying the expected data structures for both reading
+ * (converting file formats into output objects) and writing (converting input objects into file
+ * formats). This ensures proper integration between Iceberg's storage layer and processing
+ * engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return the type of the data structures handled by this model implementation
+ */
+ Class<D> type();
+
+ /**
+ * Return the schema type class for the object model implementation processed by this factory.
+ *
+ * @return the type of the schema for the data structures handled by this model implementation
+ */
+ Class<S> schemaType();
+
+ /**
+ * Creates a writer builder for data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder
+ */
+ WriteBuilder<D, S> writeBuilder(EncryptedOutputFile outputFile);
Review Comment:
Sure, I'll have a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2227809323
##########
core/src/main/java/org/apache/iceberg/io/FormatModel.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ */
+public interface FormatModel<D> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
+ *
+ * <p>The model names (for example: "generic", "spark", "spark-vectorized", "flink", "arrow") act
+ * as a contract specifying the expected data structures for both reading (converting file formats
+ * into output objects) and writing (converting input objects into file formats). This ensures
+ * proper integration between Iceberg's storage layer and processing engines.
+ *
+ * @return string identifier for this model implementation
+ */
+ String modelName();
Review Comment:
I had a hard time understanding what the model name would refer to in the passed parameters. Maybe worth making it an enum? Also would `modelType` make more sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2228068245
##########
core/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B>> {
+
+ /** Set the file schema. */
+ B fileSchema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Adds the new properties to the writer configuration.
+ *
+ * @param properties a map of writer config properties
+ * @return this for method chaining
+ */
+ B set(Map<String, String> properties);
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /**
+ * Add the new properties to file metadata for the created file.
+ *
+ * @param properties a map of file metadata properties
+ * @return this for method chaining
+ */
+ B meta(Map<String, String> properties);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Sets the encryption key used for writing the file. If the writer does not support encryption,
+ * then an exception should be thrown.
+ */
+ B fileEncryptionKey(ByteBuffer encryptionKey);
+
+ /**
+ * Sets the additional authentication data (AAD) prefix used for writing the file. If the writer
+ * does not support encryption, then an exception should be thrown.
+ */
+ B fileAADPrefix(ByteBuffer aadPrefix);
+
+ /** Sets the partition specification for the Iceberg metadata. */
+ B spec(PartitionSpec newSpec);
+
+ /** Sets the partition value for the Iceberg metadata. */
+ B partition(StructLike newPartition);
+
+ /** Sets the encryption key metadata for Iceberg metadata. */
+ B keyMetadata(EncryptionKeyMetadata metadata);
+
+ /** Sets the sort order for the Iceberg metadata. */
+ B sortOrder(SortOrder newSortOrder);
Review Comment:
these also all have the `new` prefix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2546948615
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -229,4 +206,27 @@ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> t
model != null, "Format model is not registered for format %s and type %s", format, type);
return model;
}
+
+ private static boolean equals(FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
Review Comment:
nit: I know this is INFO level. The only reason this can fail is that the class can't be found/loaded. maybe the error msg can be sth like.
```
LOG.INFO("Skip registration of {}. Likely the jar is not in the classpath", classToRegister);
```
`Unable ...` and the stack trace would sound alarming to users (although INFO level).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#issuecomment-3863962214
Thanks for the review and the merge @rdblue, and fore everybody else too!
Opened the Parquet PR: #15253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775167882
##########
core/src/main/java/org/apache/iceberg/formats/FileWriterBuilder.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriter;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <W> the concrete writer type the builder produces
+ * @param <S> the type of the schema for the input data
Review Comment:
Update for "engine schema".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2093626096
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
`ObjectModel` seems too generic and doesn't really capture the responsibility of this factory class. Some alternatives to consider `IOFactory` or `DataIOFactory`.
`FileIO` is already taken for another context. Otherwise, `FileIOFactory` may be good.
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
I see. It is odd that the symmetric counter part of `ReadBuilder` is not `WriteBuilder`.
our current class naming is a bit inconsistent in terms of `Writer` vs `Appender`. E.g. we have `ParquetWriter implements FileAppender`. In my mind, `FileAppender` should be named as `FileWriter` for write data to a specific file format. And the current `FileWriter` should be called `ContentFileWriter`, because `ContentFile` is a data, position delete, or equality delete file. But I guess we might be too late for that.
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * The name of the object model. Allows users to specify the object model to map the data file for
+ * reading and writing.
+ */
+ String name();
+
+ /**
+ * The appender builder for the output file which writes the data in the specified file format and
+ * accepts the records defined by this object model. The 'mode' parameter defines the input type
+ * for the specific writer use-cases. The appender should handle the following input in the
+ * specific modes:
+ *
+ * <ul>
+ * <li>The appender's engine specific input type
+ * <ul>
+ * <li>{@link WriteMode#DATA_WRITER}
+ * <li>{@link WriteMode#EQUALITY_DELETE_WRITER}
+ * </ul>
+ * <li>{@link org.apache.iceberg.deletes.PositionDelete} where the type of the row is the
+ * appender's engine specific input type when the 'mode' is {@link
+ * WriteMode#POSITION_DELETE_WRITER}
+ * </ul>
+ *
+ * @param outputFile to write to
+ * @param mode for the appender
+ * @return the appender builder
+ * @param <B> The type of the appender builder
+ */
+ <B extends AppenderBuilder<B, E>> B appenderBuilder(OutputFile outputFile, WriteMode mode);
Review Comment:
wondering if we need the `WriteMode` enum. Somewhere downstream, there is probably a switch case.
Instead, we can have 3 different methods like `dataWriteBuilder`, `positionDeleteWriteBuilder`, and `equalityDeleteWriteBuilder`.
If we want to stick with a enum arg, we can probably just use the existing `FileContent` enum.
##########
data/src/main/java/org/apache/iceberg/data/FileWriteBuilderBase.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+
+/**
+ * Builder for generating one of the following:
+ *
+ * <ul>
+ * <li>{@link DataWriter}
+ * <li>{@link EqualityDeleteWriter}
+ * <li>{@link PositionDeleteWriter}
+ * </ul>
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+interface FileWriteBuilderBase<B extends FileWriteBuilderBase<B, E>, E>
Review Comment:
ContentFileWriteBuilder? also, can we merge the `WriteBuildBase` into this interface too?
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If encryption is not supported by the reader
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data prefix used for writing the file. If encryption is not
Review Comment:
nit: add the acronym in the comment. When I first saw `aadPrefix`, I thought it is a typo of `addPrefix` :)
```
additional authentication data (aad)
```
##########
data/src/main/java/org/apache/iceberg/data/ObjectModelRegistry.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.AppenderBuilder;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ObjectModel;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Registry which provides the available {@link ReadBuilder}s, {@link AppenderBuilder}s and writer
+ * builders ({@link DataWriteBuilder}, {@link EqualityDeleteWriteBuilder}, {@link
+ * PositionDeleteWriteBuilder}). Based on the `file format` and the requested `object model name`
+ * the registry returns the correct reader and writer builders. These builders could be used to
+ * generate the readers and writers.
+ *
+ * <p>The available {@link ObjectModel}s are registered by the {@link
+ * #registerObjectModel(ObjectModel)} method. These {@link ObjectModel}s will be used to create the
+ * {@link ReadBuilder}s and the {@link AppenderBuilder}s. The former ones are returned directly, the
+ * later ones either returned directly or wrapped in the appropriate writer builder implementations.
+ */
+public final class ObjectModelRegistry {
Review Comment:
shouldn't this be in the same package as `ObjectModel` in iceberg-core module?
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * File formats should implement this interface to provide a builder for reading data files. {@link
+ * ReadBuilder} reads the data files with the specified parameters. The returned objects are defined
+ * by the {@link ObjectModel} which is used to read the data.
+ *
+ * <p>This interface is directly exposed for the users to parameterize readers.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ */
+public interface ReadBuilder<B extends ReadBuilder<B>> {
+ /** The configuration key for the batch size in case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the expression. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user provided setting if is not supported by them.
+ */
+ default B reuseContainers() {
+ // Skip container reuse configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata, and not coming from the data files themselves. The keys of the map are the
+ * column ids, the values are the accessors generating the values.
+ */
+ B constantFieldAccessors(Map<Integer, ?> constantFieldAccessors);
+
+ /** Sets a mapping from external schema names to Iceberg type IDs. */
+ B withNameMapping(NameMapping newNameMapping);
+
+ /**
+ * Sets the file encryption key used for reading the file. If encryption is not supported by the
+ * reader then an exception should be thrown.
+ */
+ default B withFileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data prefix for encryption. If encryption is not supported
+ * by the reader then an exception should be thrown.
+ */
+ default B withAADPrefix(ByteBuffer aadPrefix) {
Review Comment:
inconsistent naming. some methods use `with`, while some don't
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * The name of the object model. Allows users to specify the object model to map the data file for
+ * reading and writing.
+ */
+ String name();
+
+ /**
+ * The appender builder for the output file which writes the data in the specified file format and
+ * accepts the records defined by this object model. The 'mode' parameter defines the input type
+ * for the specific writer use-cases. The appender should handle the following input in the
+ * specific modes:
+ *
+ * <ul>
+ * <li>The appender's engine specific input type
+ * <ul>
+ * <li>{@link WriteMode#DATA_WRITER}
+ * <li>{@link WriteMode#EQUALITY_DELETE_WRITER}
+ * </ul>
+ * <li>{@link org.apache.iceberg.deletes.PositionDelete} where the type of the row is the
+ * appender's engine specific input type when the 'mode' is {@link
+ * WriteMode#POSITION_DELETE_WRITER}
+ * </ul>
+ *
+ * @param outputFile to write to
+ * @param mode for the appender
+ * @return the appender builder
+ * @param <B> The type of the appender builder
+ */
+ <B extends AppenderBuilder<B, E>> B appenderBuilder(OutputFile outputFile, WriteMode mode);
+
+ /**
+ * The reader builder for the input file which reads the data from the specified file format and
+ * returns the records in this object model.
+ *
+ * @param inputFile to read from
+ * @return the reader builder
+ * @param <B> The type of the reader builder
+ */
+ <B extends ReadBuilder<B>> B readBuilder(InputFile inputFile);
Review Comment:
nit: inconsistent naming: append and read (or) appender and reader.
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
+ B overwrite(boolean enabled);
+
+ /**
+ * Sets the encryption key used for writing the file. If encryption is not supported by the reader
+ * then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the additional authentication data prefix used for writing the file. If encryption is not
+ * supported by the reader then an exception should be thrown.
+ */
+ default B aadPrefix(ByteBuffer aadPrefix) {
+ throw new UnsupportedOperationException("Not supported");
+ }
+
+ /**
+ * Sets the engine native schema for the input. Defines the input type when there is N to 1
+ * mapping between the engine type and the Iceberg type, and providing the Iceberg schema is not
+ * enough for the conversion.
+ */
+ B engineSchema(E newEngineSchema);
Review Comment:
maybe `dataSchema` is better than `engineSchema`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162709629
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
+ // Skip filtering if not available
+ return (B) this;
+ }
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param newFilter the filter to set
+ */
+ default B filter(Expression newFilter) {
+ return filter(newFilter, true);
+ }
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ default B set(String key, String value) {
+ // Skip configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Enables reusing the containers returned by the reader. Decreases pressure on GC. Readers could
+ * decide to ignore the user-provided setting if it is not supported by them.
+ */
+ default B reuseContainers() {
+ // Skip container reuse configuration if not applicable
+ return (B) this;
+ }
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata and not coming from the data files themselves. The keys of the map are the column
+ * ids, the values are the accessors generating the values.
+ */
+ B constantFieldAccessors(Map<Integer, ?> constantFieldAccessors);
+
+ /** Sets a mapping from external schema names to Iceberg type IDs. */
+ B nameMapping(NameMapping newNameMapping);
+
+ /**
+ * Sets the file encryption key used for reading the file. If the reader does not support
+ * encryption, then an exception should be thrown.
+ */
+ default B fileEncryptionKey(ByteBuffer encryptionKey) {
Review Comment:
Why can't implementations throw `UnsupportedOperationException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162709047
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link ReadBuilder} instances based on:
+ *
+ * <ul>
+ * <li>source file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * </ul>
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ */
+public interface ReadBuilder<B extends ReadBuilder<B>> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Read only the given columns. */
+ B project(Schema newSchema);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some parts of the expression. In this case
+ * the reader might return unfiltered or partially filtered rows. It is the caller's
+ * responsibility to apply the filter again.
+ *
+ * @param newFilter the filter to set
+ * @param filterCaseSensitive whether the filtering is case-sensitive or not
+ */
+ default B filter(Expression newFilter, boolean filterCaseSensitive) {
Review Comment:
I don't agree that the current API is misleading, but I do understand the concern. The main problem is that this needs to change uses that currently call `caseSensitive(boolean)`. Also, if there are future cases that need to be case insensitive, I don't think it makes sense for them to each be configured individually given that the user/caller expectation is that all name matching is case sensitive or not.
For instance, other interfaces (including `Scan`) also expose `select(String... columns)` and the case sensitivity setting is used for both expression binding and for column selection. This change would set a precedent that we should instead use `select(selectCaseSensitive, String... columns)` but that would allow surprising behavior if you set `selectCaseSensitive` differently than `filterCaseSensitive`.
I think this should go back to the original `caseSensitive(boolean)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2227788782
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #registerFormatModel(FormatModel)}
+ * and used for creating readers and writers. Read builders are returned directly from the factory.
+ * Write builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void registerFormatModel(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serves %s",
Review Comment:
```suggestion
"Object model factory %s clashes with %s. Both serve %s",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2227773176
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ B split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ B project(Schema newSchema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param newCaseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean newCaseSensitive);
Review Comment:
```suggestion
B caseSensitive(boolean isCaseSensitive);
```
or maybe just `caseSensitive` without the `new` prefix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2255014461
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the input data type for the writer
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, D>, D> {
Review Comment:
Is this actually going to expose `B` to callers? It seems like `B` is purposely not exposed. If that's the case, why not remove `B`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257503105
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
Review Comment:
Moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257227208
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
+
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param start the start position for this read
+ * @param length the length of the range this read should scan
+ */
+ B split(long start, long length);
+
+ /** Set the projection schema. */
+ B project(Schema schema);
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ B caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
+ *
+ * @param filter the filter to set
+ */
+ B filter(Expression filter);
+
+ /**
+ * Sets configuration key/value pairs for the reader. Reader builders should ignore configuration
+ * keys not known for them.
+ */
+ B set(String key, String value);
+
+ /**
+ * Sets multiple configuration key/value pairs for the reader. Reader builders should ignore
+ * configuration keys not known for them.
+ */
+ @SuppressWarnings("unchecked")
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /** Enables reusing the containers returned by the reader. Decreases pressure on GC. */
+ B reuseContainers();
+
+ /**
+ * Accessors for constant field values. Used for calculating values in the result which are coming
+ * from metadata and not coming from the data files themselves. The keys of the map are the column
+ * ids, the values are the accessors generating the values.
Review Comment:
Fixed the naming and the comment.
The values need to be `?` or `Object`, because they are the fields of the rows, and not the rows itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257572102
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the input data type for the writer
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, D>, D> {
+ /** Set the file schema. */
+ B fileSchema(Schema fileSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ @SuppressWarnings("unchecked")
+ default B set(Map<String, String> properties) {
Review Comment:
For consistency's shake.
We have `meta(String, String)`, `meta(Map)`, `set(String, String)`, `set(Map)`, and now it is consistent between all of the builders
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the input data type for the writer
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, D>, D> {
+ /** Set the file schema. */
+ B fileSchema(Schema fileSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ /**
+ * Sets multiple writer configuration properties that affect the writer behavior.
+ *
+ * @param properties writer config properties to set
+ * @return this for method chaining
+ */
+ @SuppressWarnings("unchecked")
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254910767
##########
core/src/main/java/org/apache/iceberg/data/PositionDeleteWriteBuilder.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+
+/**
+ * A specialized builder for creating position-based delete file writers.
+ *
+ * <p>This builder extends the generic {@link ContentFileWriteBuilder} interface with functionality
+ * specific to creating {@link PositionDeleteWriter} instances.
+ *
+ * <p>The builder provides methods to configure the schema for the row data that might be included
+ * with the position deletes through {@link #rowSchema(Schema)}, enabling optional preservation of
+ * deleted record content.
Review Comment:
Do we need to include this in the refactor? I'm trying to think about ways to reduce scope. v3 and later require deletes to be stored as separate deletion vectors in Puffin files rather than allowing the old format. That means we don't need to do all of the work to preserve the existing functionality.
For instance, I don't think that we have had any implementations actually produce Avro/Parquet position deletes with the deleted row embedded. We also won't write stats for row content because it is much faster to assign deletes to data files by referenced data file rather than running stats comparison. So I don't think that we need to carry forward the `rowSchema` functionality.
We could consider not adding position deletes to this API at all, but I think that we want to still produce them when writing to v2 tables. I think the best choice is to support writing them, but with the simplest schema possible and not allow setting the row schema (which isn't used anyway).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2257555813
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
Review Comment:
This is an interesting question.
I didn't want to expose the specific `WriteBuilder` class, and I don't expect that the users of the method should know anything about the implementations when using the new API. OTOH, I needed to keep the possibility to call the file format specific methods to implement the old interfaces (`Parquet.WriteBuilder` and `Parquet.ReadBuilder`).
If we give up on the near seamless upgrade between the 2 APIs, and accept that we return totally different objects from the registry than previously, then we can change this behavior. But again, this is bigger than the originally intended minimal change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254967177
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.dataBuilder(outputFile.encryptingOutputFile());
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D> DataWriteBuilder<?, D> dataWriteBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ WriteBuilder<?, D> writeBuilder =
+ factory.equalityDeleteBuilder(outputFile.encryptingOutputFile());
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
Review Comment:
I think the intent of `ContentFileWriteBuilder` was to move the data- or delete-specific parts of the builder into common code that doesn't depend on the format. But this still requires the format to register builders for data, equality deletes, and position deletes. I think that this should choose one or the other.
If possible, I'd prefer the approach of making formats responsible for creating and configuring file appenders and have the data/pos-delete/eq-delete logic in shared classes. I think you pointed out elsewhere that this would require some kind of specialization or hint to tell the format to use a different set of config properties. Do you know what that would look like, besides making the format responsible for separate builders like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2254894433
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
Review Comment:
The purpose of `set` is to pass format-specific configuration through. This is configuring the object model and I think should not use a property. That's why it is a red flag that this property is defined here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2276207722
##########
core/src/main/java/org/apache/iceberg/data/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.dataBuilder(outputFile.encryptingOutputFile());
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D> DataWriteBuilder<?, D> dataWriteBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ WriteBuilder<?, D> writeBuilder =
+ factory.equalityDeleteBuilder(outputFile.encryptingOutputFile());
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
Review Comment:
Changed the current solution by:
- adding `WriteBuilder.content(FileContent)`, so the file format can chose the appropriate configuration
- adding `Function<PositionDelete<D>, D> FormatModel.positionDeleteConverter(Schema)`, so the engine can provide a converter from `PositionDelete` to the engine specific model. This allows us to use a single FileAppender to append records to the data and the position delete files as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2276209846
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the input data type for the writer
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, D>, D> {
Review Comment:
Ok.. removed this
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
Review Comment:
Removed this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2099765989
##########
data/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilderImpl.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.DeleteSchemaUtil;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.ArrayUtil;
+
+/**
+ * An internal implementation that handles all {@link ContentFileWriteBuilder} interface variants.
+ *
+ * <p>This unified implementation serves as a backend for multiple specialized content writers:
+ *
+ * <ul>
+ * <li>{@link DataWriteBuilder} for creating data files
+ * <li>{@link EqualityDeleteWriteBuilder} for creating equality delete files
+ * <li>{@link PositionDeleteWriteBuilder} for creating position delete files
+ * </ul>
+ *
+ * <p>The implementation delegates to a format-specific {@link WriteBuilder} while enriching it with
+ * content-specific functionality. When building a writer, the implementation configures the
+ * underlying builder and calls its {@link WriteBuilder#build()} method to create the appropriate
+ * specialized writer for the requested content type.
+ *
+ * @param <C> the concrete builder type for method chaining
+ * @param <W> the type of the wrapped format-specific writer builder
+ * @param <E> the engine-specific schema type required by the writer
+ */
+@SuppressWarnings("unchecked")
+class ContentFileWriteBuilderImpl<
+ C extends ContentFileWriteBuilderImpl<C, W, E>, W extends WriteBuilder<W, E>, E>
+ implements DataWriteBuilder<C, E>,
Review Comment:
@rdblue asked to separate out them as they have different configuration possibilities, and I agree with him. One doesn't need to set the `equalityFieldIds` for the `PositionDeleteWriteBuilder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2095481654
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
I needed to define a boundary for this change.
My decision was to keep the `FileAppender` (for writes), and the `CloseableIterable` (for writes) which are both widely used in the codebase.
In every other place I tried to stick to the API defined by the InternalData API where WriteBuilder and ReadBuilder was defined.
This dictates many of the choices/discrepancies you highlight here.
If we stick to the `FileAppender`, should we build it with `AppenderBuilder`, or `WriteBuilder`? Shall we change the `FileAppender` to `FileWriter` too? Shall we move it? I decided against it.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2096136844
##########
core/src/main/java/org/apache/iceberg/io/AppenderBuilder.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Interface which is implemented by the data file format implementations. The {@link ObjectModel}
+ * provides the {@link AppenderBuilder} for the given parameters:
+ *
+ * <ul>
+ * <li>file format
+ * <li>engine specific object model
+ * <li>{@link ObjectModel.WriteMode}
+ * </ul>
+ *
+ * The {@link AppenderBuilder} is used to write data to the target files.
+ *
+ * @param <B> type returned by builder API to allow chained calls
+ * @param <E> the engine specific schema of the input data
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E> {
Review Comment:
since we are doing a major refactoring of writer interface, I am wondering if there is an appetite for revisiting the current writer class organization. I just found it weird to have the opposite side of `ReadBuilder` to be `AppenderBuilder` and not `WriteBuilder`.
> FileAppender should be named as `FileWriter` for write data to a specific file format. And the current `FileWriter` should be called `ContentFileWriter` or `ContentWriter`, because it handles `ContentFile` which is a data, position delete, or equality delete file. But I guess we might be too late for that.
I don't know if we can involve this in a compatible without changing the package name of `FileWriter`. But if it is achievable, that would be really nice.
> If I remember correctly @danielcweeks had a comment suggesting to move as much as possible to data. I agree with him, as I feel that the core module starts to get bloated, and everything is pushed there. The data module already contains classes for accessing data independently of the actual File Format. I think the ObjectModelRegistry has the same function.
Agree that core module is bloated. If there is a good and compatible way to evolve the structure, I would be happy to see it.
I know it is a big effort that shouldn't be taken lightly. @aokolnychyi @danielcweeks @RussellSpitzer @rdblue any comment? maybe this can be brought up in the next community sync?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2098726421
##########
core/src/main/java/org/apache/iceberg/io/WriteBuilder.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. Each {@link
+ * FileAccessFactory} implementation provides appropriate {@link WriteBuilder} instances based on:
+ *
+ * <ul>
+ * <li>target file format (Parquet, Avro, ORC)
+ * <li>engine-specific object representation (spark, flink, generic, etc.)
+ * <li>content type ({@link FileContent#DATA}, {@link FileContent#EQUALITY_DELETES}, {@link
+ * FileContent#POSITION_DELETES})
+ * </ul>
+ *
+ * The {@link WriteBuilder} follows the builder pattern to configure and create {@link FileAppender}
+ * instances that write data to the target output files.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type for the input data records
+ */
+public interface WriteBuilder<B extends WriteBuilder<B, E>, E> {
+ /** Set the file schema. */
+ B schema(Schema newSchema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
+ *
+ * @param property a writer config property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B set(String property, String value);
+
+ default B set(Map<String, String> properties) {
+ properties.forEach(this::set);
+ return (B) this;
+ }
+
+ /**
+ * Set a file metadata property in the created file.
+ *
+ * @param property a file metadata property name
+ * @param value config value
+ * @return this for method chaining
+ */
+ B meta(String property, String value);
+
+ /** Sets the metrics configuration used for collecting column metrics for the created file. */
+ B metricsConfig(MetricsConfig newMetricsConfig);
+
+ /** Overwrite the file if it already exists. By default, overwrite is disabled. */
+ B overwrite();
+
+ /**
+ * Overwrite the file if it already exists. The default value is <code>false</code>.
+ *
+ * @deprecated Since 1.10.0, will be removed in 1.11.0. Only provided for backward compatibility.
+ * Use {@link #overwrite()} instead.
+ */
+ @Deprecated
Review Comment:
Why are we adding in a deprecated method to a new interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2130530826
##########
data/src/main/java/org/apache/iceberg/data/ContentFileWriteBuilder.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.WriteBuilder;
+
+/**
+ * A generic builder interface for creating specialized file writers in the Iceberg ecosystem.
+ *
+ * <p>This builder provides a unified configuration API for generating various types of content
+ * writers:
+ *
+ * <ul>
+ * <li>{@link DataWriter} for creating data files with table records
+ * <li>{@link EqualityDeleteWriter} for creating files with equality-based delete records
+ * <li>{@link PositionDeleteWriter} for creating files with position-based delete records
+ * </ul>
+ *
+ * <p>Each concrete implementation configures the underlying file format writer while adding
+ * content-specific metadata and behaviors.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <E> engine-specific schema type required by the writer for data conversion
+ */
+interface ContentFileWriteBuilder<B extends ContentFileWriteBuilder<B, E>, E> {
Review Comment:
yeah. I agree. There are duplicates because `FileWriter` would encapsulate `FileAppender` object. Hence `FileWriterBuilder` need to forward those to the `FileAppenderBuilder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2340485143
##########
core/src/main/java/org/apache/iceberg/io/FormatModel.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.util.function.Function;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.FormatModelRegistry;
+import org.apache.iceberg.deletes.PositionDelete;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Returns the unique identifier for the object model implementation processed by this factory.
+ *
+ * <p>The model names (for example: "generic", "spark", "spark-vectorized", "flink", "arrow") act
+ * as a contract specifying the expected data structures for both reading (converting file formats
+ * into output objects) and writing (converting input objects into file formats). This ensures
+ * proper integration between Iceberg's storage layer and processing engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return string identifier for this model implementation
+ */
+ Class<D> type();
+
+ /**
+ * Creates a writer builder for standard data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
Review Comment:
Removed most of the doc. Kept only a minimal part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2162710191
##########
core/src/main/java/org/apache/iceberg/io/ReadBuilder.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FileAccessFactory} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ *
+ * @param <B> the concrete builder type for method chaining
+ * @param <D> the output data type produced by the reader
+ */
+public interface ReadBuilder<B extends ReadBuilder<B, D>, D> {
+ /** The configuration key for the batch size in the case of vectorized reads. */
+ String RECORDS_PER_BATCH_KEY = "iceberg.records-per-batch";
Review Comment:
This doesn't appear to be used and also should not be a on the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2222665248
##########
core/src/main/java/org/apache/iceberg/io/ObjectModel.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.FileFormat;
+
+/**
+ * Direct conversion is used between file formats and engine internal formats for performance
+ * reasons. Object models encapsulate these conversions.
+ *
+ * <p>{@link ReadBuilder} is provided for reading data files stored in a given {@link FileFormat}
+ * into the engine specific object model.
+ *
+ * <p>{@link AppenderBuilder} is provided for writing engine specific object model to data/delete
+ * files stored in a given {@link FileFormat}.
+ *
+ * <p>Iceberg supports the following object models natively:
+ *
+ * <ul>
+ * <li>generic - reads and writes Iceberg {@link org.apache.iceberg.data.Record}s
+ * <li>spark - reads and writes Spark InternalRow records
+ * <li>spark-vectorized - vectorized reads for Spark columnar batches. Not supported for {@link
+ * FileFormat#AVRO}
+ * <li>flink - reads and writes Flink RowData records
+ * <li>arrow - vectorized reads for into Arrow columnar format. Only supported for {@link
+ * FileFormat#PARQUET}
+ * </ul>
+ *
+ * <p>Engines could implement their own object models to leverage Iceberg data file reading and
+ * writing capabilities.
+ *
+ * @param <E> the engine specific schema of the input data for the appender
+ */
+public interface ObjectModel<E> {
Review Comment:
Also put together a solution where the Engine specific format transformation is separated from the writer, and the engines need to take care of it separately.
This is somewhat complicated on the implementation side (see: [RowDataTransformer](https://github.com/apache/iceberg/pull/12298/files#diff-562fa4cc369c908a157f59a9235fd3f389096451e7901686fba37c87b53dee08), and [InternalRowTransformer](https://github.com/apache/iceberg/pull/12298/files#diff-546f9dc30e3207d1d2bc0a2722976b55f5a04dcf85a22855e4f400500c317140)), but simplifies the API.
@rdblue: Please check the proposed solution. I think this is what you have suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2068340171
##########
data/src/main/java/org/apache/iceberg/data/AppenderBuilder.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder for generating a {@link FileAppender}.
+ *
+ * @param <B> type of the builder
+ * @param <E> engine specific schema of the input records used for appender initialization
+ */
+public interface AppenderBuilder<B extends AppenderBuilder<B, E>, E>
Review Comment:
There are a few convenience methods in the `data.AppenderBuilder`, like `B set(Map<String, String> properties)`, or `B meta(Map<String, String> properties)` what we don't enforce for the `io.AppenderBuilder` to implement, but these could be handled by default methods.
The main thing is that we don't want to expose the `<D> FileAppender<D> build(WriteMode mode) throws IOException;` to the data writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2726954120
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
Review Comment:
You can do this by calling:
```
public static synchronized void register(FormatModel<?, ?> formatModel);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444540480
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
Review Comment:
Removed
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> MODELS =
+ Maps.newConcurrentMap();
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#type()}
+ */
+ public static void register(FormatModel<?, ?> formatModel) {
+ Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type());
+
+ FormatModel<?, ?> existing = MODELS.get(key);
+ Preconditions.checkArgument(
+ existing == null || checkFormatModelEquals(existing, formatModel),
+ "Cannot register %s: %s is registered for format=%s type=%s schemaType=%s",
+ formatModel.getClass(),
+ existing == null ? null : existing.getClass(),
+ key.first(),
+ key.second(),
+ existing == null ? null : existing.schemaType());
+
+ MODELS.put(key, formatModel);
+ }
+
+ private static boolean checkFormatModelEquals(
+ FormatModel<?, ?> model1, FormatModel<?, ?> model2) {
+ return Objects.equals(model1.getClass().getName(), model2.getClass().getName())
+ && Objects.equals(model1.type().getName(), model2.type().getName())
+ && Objects.equals(
+ model1.schemaType() == null ? null : model1.schemaType().getName(),
+ model2.schemaType() == null ? null : model2.schemaType().getName());
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {}: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param type the output type
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D, S> ReadBuilder readBuilder(
+ FileFormat format, Class<D> type, InputFile inputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D, S> DataWriteBuilder<D, S> dataWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder writeBuilder =
+ factory.writeBuilder(outputFile.encryptingOutputFile()).content(FileContent.DATA);
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with equality deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer persists equality delete
+ * records that identify rows to be deleted based on the configured equality fields, producing a
+ * {@link DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param type the input type
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @param <S> the type of the input schema for the writer
+ * @return a configured delete write builder for creating an {@link EqualityDeleteWriter}
+ */
+ public static <D, S> EqualityDeleteWriteBuilder<D, S> equalityDeleteWriteBuilder(
+ FileFormat format, Class<D> type, EncryptedOutputFile outputFile) {
+ FormatModel<D, S> factory = factoryFor(format, type);
+ WriteBuilder writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.EQUALITY_DELETES);
+ return ContentFileWriteBuilderImpl.forEqualityDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ /**
+ * Creates a writer builder for generating a {@link DeleteFile} with position-based deletes.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the given file format. The writer accepts {@link PositionDelete}
+ * records that identify rows to be deleted by file path and position, producing a {@link
+ * DeleteFile} that can be used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param outputFile destination for the written data
+ * @return a configured delete write builder for creating a {@link PositionDeleteWriter}
+ */
+ public static PositionDeleteWriteBuilder positionDeleteWriteBuilder(
+ FileFormat format, EncryptedOutputFile outputFile) {
+ FormatModel<PositionDelete<?>, ?> factory = factoryForPositionDelete(format);
+ WriteBuilder writeBuilder =
+ factory
+ .writeBuilder(outputFile.encryptingOutputFile())
+ .content(FileContent.POSITION_DELETES);
+ return ContentFileWriteBuilderImpl.forPositionDelete(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
+ }
+
+ @VisibleForTesting
+ static Map<Pair<FileFormat, Class<?>>, FormatModel<?, ?>> models() {
+ return MODELS;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <D, S> FormatModel<D, S> factoryFor(FileFormat format, Class<D> type) {
+ FormatModel<D, S> model = ((FormatModel<D, S>) MODELS.get(Pair.of(format, type)));
+ Preconditions.checkArgument(
+ model != null, "Format model is not registered for %s and %s", format, type);
+ return model;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static FormatModel<PositionDelete<?>, ?> factoryForPositionDelete(FileFormat format) {
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444547360
##########
core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.common.DynMethods;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDelete;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.FormatModel;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.ReadBuilder;
+import org.apache.iceberg.io.WriteBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A registry that manages file-format-specific readers and writers through a unified object model
+ * factory interface.
+ *
+ * <p>This registry provides access to {@link ReadBuilder}s for data consumption and various writer
+ * builders:
+ *
+ * <ul>
+ * <li>{@link WriteBuilder} for basic file writing,
+ * <li>{@link DataWriteBuilder} for data files,
+ * <li>{@link EqualityDeleteWriteBuilder} for equality deletes,
+ * <li>{@link PositionDeleteWriteBuilder} for position deletes.
+ * </ul>
+ *
+ * The appropriate builder is selected based on {@link FileFormat} and object model name.
+ *
+ * <p>{@link FormatModel} objects are registered through {@link #register(FormatModel)} and used for
+ * creating readers and writers. Read builders are returned directly from the factory. Write
+ * builders may be wrapped in specialized content file writer implementations depending on the
+ * requested builder type.
+ */
+public final class FormatModelRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(FormatModelRegistry.class);
+ // The list of classes which are used for registering the reader and writer builders
+ private static final List<String> CLASSES_TO_REGISTER = ImmutableList.of();
+
+ private static final Map<Pair<FileFormat, String>, FormatModel<?>> FORMAT_MODELS =
+ Maps.newConcurrentMap();
+
+ /**
+ * Registers an {@link FormatModel} in this registry.
+ *
+ * <p>The {@link FormatModel} creates readers and writers for a specific combinations of file
+ * format (Parquet, ORC, Avro) and object model (for example: "generic", "spark", "flink", etc.).
+ * Registering custom factories allows integration of new data processing engines for the
+ * supported file formats with Iceberg's file access mechanisms.
+ *
+ * <p>Each factory must be uniquely identified by its combination of file format and object model
+ * name. This uniqueness constraint prevents ambiguity when selecting factories for read and write
+ * operations.
+ *
+ * @param formatModel the factory implementation to register
+ * @throws IllegalArgumentException if a factory is already registered for the combination of
+ * {@link FormatModel#format()} and {@link FormatModel#modelName()}
+ */
+ @SuppressWarnings("CatchBlockLogException")
+ public static void register(FormatModel<?> formatModel) {
+ Pair<FileFormat, String> key = Pair.of(formatModel.format(), formatModel.modelName());
+ if (FORMAT_MODELS.containsKey(key)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Object model factory %s clashes with %s. Both serve %s",
+ formatModel.getClass(), FORMAT_MODELS.get(key), key));
+ }
+
+ FORMAT_MODELS.put(key, formatModel);
+ }
+
+ @SuppressWarnings("CatchBlockLogException")
+ private static void registerSupportedFormats() {
+ // Uses dynamic methods to call the `register` for the listed classes
+ for (String classToRegister : CLASSES_TO_REGISTER) {
+ try {
+ DynMethods.builder("register").impl(classToRegister).buildStaticChecked().invoke();
+ } catch (NoSuchMethodException e) {
+ // failing to register a factory is normal and does not require a stack trace
+ LOG.info("Unable to register {} for data files: {}", classToRegister, e.getMessage());
+ }
+ }
+ }
+
+ static {
+ registerSupportedFormats();
+ }
+
+ private FormatModelRegistry() {}
+
+ /**
+ * Returns a reader builder for the specified file format and object model.
+ *
+ * <p>The returned {@link ReadBuilder} provides a fluent interface for configuring how data is
+ * read from the input file and converted to the output objects. The builder supports
+ * configuration options like schema projection, predicate pushdown, batch size and encryption.
+ *
+ * @param format the file format (Parquet, Avro, ORC) that determines the parsing implementation
+ * @param modelName identifier for the expected output data representation (generic, spark, flink,
+ * etc.)
+ * @param inputFile source file to read data from
+ * @param <D> the type of data records the reader will produce
+ * @return a configured reader builder for the specified format and object model
+ */
+ public static <D> ReadBuilder<?, D> readBuilder(
+ FileFormat format, String modelName, InputFile inputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.readBuilder(inputFile);
+ }
+
+ /**
+ * Returns a writer builder for appending data to the specified output file.
+ *
+ * <p>The returned builder produces a {@link FileAppender} that accepts records defined by the
+ * specified object model and persists them using the given file format. Data is written to the
+ * output file, but this basic writer does not collect or return {@link ContentFile} metadata.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured writer builder for creating the appender
+ */
+ public static <D> WriteBuilder<?, D> writeBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ return factory.dataBuilder(outputFile.encryptingOutputFile());
+ }
+
+ /**
+ * Returns a writer builder for generating a {@link DataFile}.
+ *
+ * <p>The returned builder produces a writer that accepts records defined by the specified object
+ * model and persists them using the provided file format. Unlike basic writers, this writer
+ * collects file metadata during the writing process and generates a {@link DataFile} that can be
+ * used for table operations.
+ *
+ * @param format the file format used for writing
+ * @param modelName name of the object model defining the input format
+ * @param outputFile destination for the written data
+ * @param <D> the type of data records the writer will accept
+ * @return a configured data write builder for creating a {@link DataWriter}
+ */
+ public static <D> DataWriteBuilder<?, D> dataWriteBuilder(
+ FileFormat format, String modelName, EncryptedOutputFile outputFile) {
+ FormatModel<D> factory = factoryFor(format, modelName);
+ WriteBuilder<?, D> writeBuilder =
+ factory.equalityDeleteBuilder(outputFile.encryptingOutputFile());
+ return ContentFileWriteBuilderImpl.forDataFile(
+ writeBuilder, outputFile.encryptingOutputFile().location(), format);
Review Comment:
This is solved now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2444538443
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
+
+ /**
+ * Set a writer configuration property which affects the writer behavior.
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/formats/ReadBuilder.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.mapping.NameMapping;
+
+/**
+ * Builder interface for creating file readers across supported data file formats. The {@link
+ * FormatModel} implementations provides appropriate {@link ReadBuilder} instances
+ *
+ * <p>The {@link ReadBuilder} follows the builder pattern to configure and create {@link
+ * CloseableIterable} instances that read data from source files. Configuration options include
+ * schema projection, predicate filtering, record batching, and encryption settings.
+ *
+ * <p>This interface is directly exposed to users for parameterizing readers.
+ */
+public interface ReadBuilder {
+ /**
+ * Restricts the read to the given range: [start, start + length).
+ *
+ * @param newStart the start position for this read
+ * @param newLength the length of the range this read should scan
+ */
+ ReadBuilder split(long newStart, long newLength);
+
+ /** Set the projection schema. */
+ ReadBuilder project(Schema schema);
+
+ /** Sets the expected output schema. If not provided derived from the {@link #project(Schema)}. */
+ default ReadBuilder outputSchema(Object schema) {
+ throw new UnsupportedOperationException("Not implemented yet");
+ }
+
+ /**
+ * Configures whether filtering should be case-sensitive. If the reader supports filtering, it
+ * must respect this setting.
+ *
+ * @param caseSensitive indicates if filtering is case-sensitive
+ */
+ ReadBuilder caseSensitive(boolean caseSensitive);
+
+ /**
+ * Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
+ * records. Some readers might not be able to filter some part of the exception. In this case the
+ * reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
+ * apply the filter again. The default implementation sets the filter to be case-sensitive.
Review Comment:
Moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2499413055
##########
core/src/main/java/org/apache/iceberg/formats/FormatModel.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.InputFile;
+
+/**
+ * Interface that provides a unified abstraction for converting between data file formats and
+ * input/output data representations.
+ *
+ * <p>{@link FormatModel} serves as a bridge between storage formats ({@link FileFormat}) and
+ * expected input/output data structures, optimizing performance through direct conversion without
+ * intermediate representations. File format implementations handle the low-level parsing details
+ * while the object model determines the in-memory representation used for the parsed data.
+ * Together, these provide a consistent API for consuming data files while optimizing for specific
+ * processing engines.
+ *
+ * <p>Iceberg provides some built-in object models and processing engines can implement custom
+ * object models to integrate with Iceberg's file reading and writing capabilities.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ */
+public interface FormatModel<D, S> {
+ /** The file format which is read/written by the object model. */
+ FileFormat format();
+
+ /**
+ * Return the row type class for the object model implementation processed by this factory.
+ *
+ * <p>The model types act as a contract specifying the expected data structures for both reading
+ * (converting file formats into output objects) and writing (converting input objects into file
+ * formats). This ensures proper integration between Iceberg's storage layer and processing
+ * engines.
+ *
+ * <p>Processing engines can define their own object models by implementing this interface and
+ * using their own model name. They can register these models with Iceberg by using the {@link
+ * FormatModelRegistry}. This allows custom data representations to be seamlessly integrated with
+ * Iceberg's file format handlers.
+ *
+ * @return the type of the data structures handled by this model implementation
+ */
+ Class<D> type();
+
+ /**
+ * Return the schema type class for the object model implementation processed by this factory.
+ *
+ * @return the type of the schema for the data structures handled by this model implementation
+ */
+ Class<S> schemaType();
+
+ /**
+ * Creates a writer builder for data files.
+ *
+ * <p>The returned {@link WriteBuilder} configures and creates a writer that converts input
+ * objects into the file format supported by this factory.
+ *
+ * @param outputFile destination for the written data
+ * @return configured writer builder
+ */
+ WriteBuilder<D, S> writeBuilder(EncryptedOutputFile outputFile);
Review Comment:
`EncryptedOutputFile` and `OutputFile` are mostly interchangeableābut not everywhere.
The **`FileWriterFactory`** API **expects an `EncryptedOutputFile`** as input. In `FormatModelRegistry`, we can obtain an `OutputFile` using `EncryptedOutputFile.encryptingOutputFile()`. However, this means we canāt pass the original `EncryptedOutputFile` down to `Parquet.write`, which looks like this:
https://github.com/apache/iceberg/blob/4e68ff008002eeb323c0c90321f6fbf867b02aa2/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L148-L157
If we pass the result of `EncryptedOutputFile.encryptingOutputFile()`, the following code path is used:
https://github.com/apache/iceberg/blob/4e68ff008002eeb323c0c90321f6fbf867b02aa2/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L140-L146
Since `EncryptedOutputFile.encryptingOutputFile()` returns a plain `OutputFile`, the file will not be encrypted.
To make the `FormatModel` API symmetric, we would need to change:
```
ReadBuilder<D, S> readBuilder(InputFile inputFile);
```
to
```
ReadBuilder<D, S> readBuilder(EncryptedInputFile inputFile);
```
But that feels unnecessary and doesnāt add real value.
@ggershinsky: What's your take? Will we need to push `EncryptedInputFile` to the file format readers sometime in the future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2775973857
##########
core/src/main/java/org/apache/iceberg/formats/BaseFormatModel.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.util.Map;
+import org.apache.iceberg.Schema;
+
+/**
+ * Base implementation of {@link FormatModel} that provides common functionality for format models.
+ *
+ * <p>This abstract class serves as a foundation for creating format-specific models that handle
+ * reading and writing data in various file formats.
+ *
+ * @param <D> output type used for reading data, and input type for writing data and deletes
+ * @param <S> the type of the schema for the input/output data
+ * @param <W> the writer type produced by the writer function
+ * @param <R> the reader type produced by the reader function
+ * @param <F> the file schema type used by the underlying file format
+ */
+public abstract class BaseFormatModel<D, S, W, R, F> implements FormatModel<D, S> {
+ private final Class<? extends D> type;
+ private final Class<S> schemaType;
+ private final WriterFunction<W, S, F> writerFunction;
+ private final ReaderFunction<R, S, F> readerFunction;
+
+ /**
+ * Constructs a new BaseFormatModel with the specified configuration.
+ *
+ * @param type the row type class for the object model implementation processed by this factory.
+ * @param schemaType the schema type class for the object model implementation processed by this
+ * factory.
+ * @param writerFunction the function used to create writers for this format
+ * @param readerFunction the function used to create readers for this format
+ */
+ protected BaseFormatModel(
+ Class<? extends D> type,
+ Class<S> schemaType,
+ WriterFunction<W, S, F> writerFunction,
+ ReaderFunction<R, S, F> readerFunction) {
+ this.type = type;
+ this.schemaType = schemaType;
+ this.writerFunction = writerFunction;
+ this.readerFunction = readerFunction;
+ }
+
+ @Override
+ public Class<? extends D> type() {
+ return type;
+ }
+
+ @Override
+ public Class<S> schemaType() {
+ return schemaType;
+ }
+
+ /**
+ * Returns the writer function used to create writers for this format.
+ *
+ * @return the writer function
+ */
+ protected WriterFunction<W, S, F> writerFunction() {
+ return writerFunction;
+ }
+
+ /**
+ * Returns the reader function used to create readers for this format.
+ *
+ * @return the reader function
+ */
+ protected ReaderFunction<R, S, F> readerFunction() {
+ return readerFunction;
+ }
+
+ /**
+ * A functional interface for creating writers that can write data in a specific format.
+ *
+ * @param <W> the writer type to be created
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#discussion_r2457581909
##########
core/src/main/java/org/apache/iceberg/formats/WriteBuilder.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.formats;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Map;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.io.FileAppender;
+
+/**
+ * Builder interface for creating file writers across supported data file formats. The {@link
+ * FormatModel} implementations provide the appropriate {@link WriteBuilder} instances.
+ *
+ * <p>The {@link WriteBuilder} follows the builder pattern to configure and create {@link
+ * FileAppender} instances that write data to the target output files.
+ *
+ * <p>This interface is directly exposed to users for parameterizing when only an appender is
+ * required.
+ */
+public interface WriteBuilder {
+ /** Set the file schema. */
+ WriteBuilder schema(Schema schema);
+
+ /**
+ * Sets the input schema accepted by the writer. If not provided derived from the {@link
+ * #schema(Schema)}.
+ */
+ WriteBuilder inputSchema(Object schema);
Review Comment:
I'm not sure why this is `Object`. I thought that the builders would be parameterized by this type. Unlike the Parquet and Avro builders, the `DataWriteBuilder` and delete builders do not expose the `writerFunc` method of building a writer tree. Instead, those builders accept the Iceberg schema and engine schema and configure the Avro and Parquet builders by calling `writerFunc`. So the new builders should have a type parameter for this, but the wrapped format-specific builders should not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data: File Format API interfaces [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #12774:
URL: https://github.com/apache/iceberg/pull/12774#issuecomment-3853441434
Merged the changes from #12298.
The remaining open questions are highlighted:
- A potential (though unlikely) breaking change: https://github.com/apache/iceberg/pull/12774#discussion_r2768937825
- Castingārelated ugliness: https: https://github.com/apache/iceberg/pull/12774#discussion_r2768968383
Based on @rdblueās comment on the other PR, the change is āabout ready to go in.ā
However, as it has changed somewhat compared to earlier iterations, previous reviewers may want to take a final look.
Thanks for everyone spending time on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org