You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2024/04/09 02:21:09 UTC

[PR] [core] Introduce file index read/write framework. [paimon]

leaves12138 opened a new pull request, #3177:
URL: https://github.com/apache/paimon/pull/3177

   
   * API:
   
   ```sql
   CREATE TABLE <PAIMON_TABLE> (<COLUMN> <COLUMN_TYPE> , ...) WITH
   (
   "index.columns" = "c1:bloom:{items=200, fpp=0.1}; c2:bloom:{items=1000000, fpp=0.03}"
   )
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3177:
URL: https://github.com/apache/paimon/pull/3177#discussion_r1565447211


##########
paimon-core/src/main/java/org/apache/paimon/operation/RawFileSplitRead.java:
##########
@@ -121,26 +127,38 @@ public RecordReader<InternalRow> createReader(DataSplit split) throws IOExceptio
 
         for (DataFileMeta file : split.dataFiles()) {
             String formatIdentifier = DataFilePathFactory.formatIdentifier(file.fileName());
-            BulkFormatMapping bulkFormatMapping =
+            RawFileBulkFormatMapping bulkFormatMapping =
                     bulkFormatMappings.computeIfAbsent(
                             new FormatKey(file.schemaId(), formatIdentifier),
                             this::createBulkFormatMapping);
 
             BinaryRow partition = split.partition();
-            suppliers.add(
+            ConcatRecordReader.ReaderSupplier<InternalRow> supplier =
                     () ->
                             createFileReader(
                                     partition,
                                     file,
                                     dataFilePathFactory,
                                     bulkFormatMapping,
-                                    dvFactory));
+                                    dvFactory);
+
+            suppliers.add(
+                    fileIndexReadEnabled

Review Comment:
   Can be in `createFileReader`?



##########
paimon-core/src/main/java/org/apache/paimon/io/FileIndexFileReader.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.io;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.fileindex.FileIndexPredicate;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.mergetree.compact.ConcatRecordReader;
+import org.apache.paimon.predicate.Predicate;
+import org.apache.paimon.predicate.PredicateBuilder;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.schema.TableSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** File index reader, do the filter in the constructor. */
+public class FileIndexFileReader implements RecordReader<InternalRow> {

Review Comment:
   FileIndexRecordReader



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3177:
URL: https://github.com/apache/paimon/pull/3177#discussion_r1562382640


##########
paimon-common/src/main/java/org/apache/paimon/utils/BloomFilter64.java:
##########
@@ -81,4 +82,45 @@ public int getNumHashFunctions() {
     public BitSet getBitSet() {
         return bitSet;
     }
+
+    /** Bit set used for bloom filter 64. */
+    public static class BitSet {
+
+        public final long[] data;

Review Comment:
   Just use byte[]?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 closed pull request #3177: [core] Introduce file index read/write framework.
URL: https://github.com/apache/paimon/pull/3177


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #3177:
URL: https://github.com/apache/paimon/pull/3177#issuecomment-2056543225

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3177:
URL: https://github.com/apache/paimon/pull/3177#discussion_r1561929554


##########
paimon-core/src/main/java/org/apache/paimon/io/DataFileMeta.java:
##########
@@ -83,6 +84,9 @@ public class DataFileMeta {
     // We have to keep the compatibility.
     private final @Nullable Long deleteRowCount;
 
+    // file index filter bytes, if it is small, store in data file meta
+    private final BinaryRow filter;

Review Comment:
   it is a file.... it should be `byte[]`, and it should be nullable



##########
paimon-core/src/main/java/org/apache/paimon/io/DataFileMeta.java:
##########
@@ -83,6 +84,9 @@ public class DataFileMeta {
     // We have to keep the compatibility.
     private final @Nullable Long deleteRowCount;
 
+    // file index filter bytes, if it is small, store in data file meta
+    private final BinaryRow filter;

Review Comment:
   embeddedIndex?



##########
paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java:
##########
@@ -100,7 +104,9 @@ public AppendOnlyWriter(
             String fileCompression,
             String spillCompression,
             FieldStatsCollector.Factory[] statsCollectors,
-            MemorySize maxDiskSize) {
+            MemorySize maxDiskSize,
+            Map<String, Map<String, Options>> fileIndexes,
+            long indexSizeInMeta) {

Review Comment:
   change this name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3177:
URL: https://github.com/apache/paimon/pull/3177#discussion_r1565131428


##########
paimon-core/src/main/java/org/apache/paimon/io/IndexWriter.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.io;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.fileindex.FileIndexFormat;
+import org.apache.paimon.fileindex.FileIndexer;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Pair;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** Index file writer. */
+public final class IndexWriter {

Review Comment:
   FileIndexWriter



##########
paimon-core/src/main/java/org/apache/paimon/io/IndexWriter.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.io;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.fileindex.FileIndexFormat;
+import org.apache.paimon.fileindex.FileIndexer;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Pair;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** Index file writer. */
+public final class IndexWriter {
+
+    private static final Pair<byte[], List<String>> EMPTY_RESULT =
+            Pair.of(null, Collections.emptyList());
+
+    private final FileIO fileIO;
+
+    private final Path path;
+
+    // if the filter size greater than fileIndexInManifestThreshold, we put it in file
+    private final long inManifestThreshold;
+
+    private final List<IndexMaintainer> indexMaintainers = new ArrayList<>();
+
+    private final List<String> resultFileNames = new ArrayList<>();
+
+    private byte[] embeddedIndexBytes;
+
+    public IndexWriter(
+            FileIO fileIO,
+            Path path,
+            RowType rowType,
+            Map<String, Map<String, Options>> fileIndexes,
+            long inManifestThreshold) {
+        this.fileIO = fileIO;
+        this.path = path;
+        this.inManifestThreshold = inManifestThreshold;
+        List<DataField> fields = rowType.getFields();
+        Map<String, DataField> map = new HashMap<>();
+        Map<String, Integer> index = new HashMap<>();
+        fields.forEach(
+                dataField -> {
+                    map.put(dataField.name(), dataField);
+                    index.put(dataField.name(), rowType.getFieldIndex(dataField.name()));
+                });
+        for (Map.Entry<String, Map<String, Options>> entry : fileIndexes.entrySet()) {
+            String columnName = entry.getKey();
+            DataField field = map.get(columnName);
+            if (field == null) {
+                throw new IllegalArgumentException(columnName + " does not exist in column fields");
+            }
+            for (Map.Entry<String, Options> typeEntry : entry.getValue().entrySet()) {
+                String indexType = typeEntry.getKey();
+                indexMaintainers.add(
+                        new IndexMaintainer(
+                                columnName,
+                                indexType,
+                                FileIndexer.create(indexType, field.type(), typeEntry.getValue())
+                                        .createWriter(),
+                                InternalRow.createFieldGetter(
+                                        field.type(), index.get(columnName))));
+            }
+        }
+    }
+
+    public void write(InternalRow row) {
+        indexMaintainers.forEach(indexMaintainer -> indexMaintainer.write(row));
+    }
+
+    public void close() throws IOException {
+        if (indexMaintainers.isEmpty()) {
+            return;
+        }
+
+        Map<String, Map<String, byte[]>> indexMaps = new HashMap<>();
+
+        for (IndexMaintainer indexMaintainer : indexMaintainers) {
+            indexMaps
+                    .computeIfAbsent(indexMaintainer.getColumnName(), k -> new HashMap<>())
+                    .put(indexMaintainer.getIndexType(), indexMaintainer.serializedBytes());
+        }
+
+        ByteArrayOutputStream baos = new ByteArrayOutputStream();
+        try (FileIndexFormat.Writer writer = FileIndexFormat.createWriter(baos)) {
+            writer.writeColumnIndexes(indexMaps);
+        }
+
+        if (baos.size() > inManifestThreshold) {
+            try (OutputStream outputStream = fileIO.newOutputStream(path, false)) {
+                outputStream.write(baos.toByteArray());
+            }
+            resultFileNames.add(path.getName());
+        } else {
+            embeddedIndexBytes = baos.toByteArray();
+        }
+    }
+
+    public Pair<byte[], List<String>> result() {

Review Comment:
   return a `FileIndexResult` {
         @Nullable byte[] embeddedIndexBytes();
         @Nullable String independentIndexFile();
   }



##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -34,6 +34,10 @@ public class DataFilePathFactory {
 
     public static final String CHANGELOG_FILE_PREFIX = "changelog-";
 
+    public static final String INDEX_PATH_PREFIX = "index-";

Review Comment:
   remove prefix



##########
paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java:
##########
@@ -78,6 +80,8 @@ public class AppendOnlyWriter implements RecordWriter<InternalRow>, MemoryOwner
     private SinkWriter sinkWriter;
     private final FieldStatsCollector.Factory[] statsCollectors;
     private final IOManager ioManager;
+    private final Map<String, Map<String, Options>> fileIndexes;

Review Comment:
   Extract a class `FileIndexOptions` to wrap two fields.



##########
paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java:
##########
@@ -70,6 +74,12 @@ public String uuid() {
         return uuid;
     }
 
+    public static Path toIndexPath(Path filePath) {

Review Comment:
   toFileIndexPath



##########
paimon-core/src/main/java/org/apache/paimon/io/IndexMaintainer.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.io;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.fileindex.FileIndexWriter;
+
+/** One index maintainer for one column. */
+public class IndexMaintainer {

Review Comment:
   `FileIndexMaintainer`
   
   Embedded it to `FileIndexWriter`.



##########
paimon-core/src/main/java/org/apache/paimon/io/RowDataFileWriter.java:
##########
@@ -66,24 +76,38 @@ public RowDataFileWriter(
         this.schemaId = schemaId;
         this.seqNumCounter = seqNumCounter;
         this.statsArraySerializer = new FieldStatsArraySerializer(writeSchema);
+        this.indexWriter =
+                new IndexWriter(

Review Comment:
   @Nullable IndexWriter.create 



##########
paimon-core/src/main/java/org/apache/paimon/io/IndexWriter.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.io;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.fileindex.FileIndexFormat;
+import org.apache.paimon.fileindex.FileIndexer;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Pair;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** Index file writer. */
+public final class IndexWriter {

Review Comment:
   implement Closeable



##########
paimon-core/src/main/java/org/apache/paimon/io/RowDataFileWriter.java:
##########
@@ -43,6 +50,7 @@ public class RowDataFileWriter extends StatsCollectingSingleFileWriter<InternalR
     private final long schemaId;
     private final LongCounter seqNumCounter;
     private final FieldStatsArraySerializer statsArraySerializer;
+    private final IndexWriter indexWriter;

Review Comment:
   @Nullable



##########
paimon-core/src/main/java/org/apache/paimon/operation/AppendOnlyFileStoreScan.java:
##########
@@ -31,15 +32,24 @@
 import org.apache.paimon.types.RowType;
 import org.apache.paimon.utils.SnapshotManager;
 
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 /** {@link FileStoreScan} for {@link AppendOnlyFileStore}. */
 public class AppendOnlyFileStoreScan extends AbstractFileStoreScan {
 
     private final FieldStatsConverters fieldStatsConverters;
 
+    private final boolean fileIndexReadEnabled;
+
     private Predicate filter;
 
+    private final Map<Long, Predicate> dataFilterMapping = new HashMap<>();

Review Comment:
   comment: just cache.



##########
paimon-core/src/main/java/org/apache/paimon/io/IndexWriter.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.io;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.fileindex.FileIndexFormat;
+import org.apache.paimon.fileindex.FileIndexer;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Pair;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** Index file writer. */
+public final class IndexWriter {
+
+    private static final Pair<byte[], List<String>> EMPTY_RESULT =
+            Pair.of(null, Collections.emptyList());
+
+    private final FileIO fileIO;
+
+    private final Path path;
+
+    // if the filter size greater than fileIndexInManifestThreshold, we put it in file
+    private final long inManifestThreshold;
+
+    private final List<IndexMaintainer> indexMaintainers = new ArrayList<>();
+
+    private final List<String> resultFileNames = new ArrayList<>();
+
+    private byte[] embeddedIndexBytes;
+
+    public IndexWriter(
+            FileIO fileIO,
+            Path path,
+            RowType rowType,
+            Map<String, Map<String, Options>> fileIndexes,
+            long inManifestThreshold) {
+        this.fileIO = fileIO;
+        this.path = path;
+        this.inManifestThreshold = inManifestThreshold;
+        List<DataField> fields = rowType.getFields();
+        Map<String, DataField> map = new HashMap<>();
+        Map<String, Integer> index = new HashMap<>();
+        fields.forEach(
+                dataField -> {
+                    map.put(dataField.name(), dataField);
+                    index.put(dataField.name(), rowType.getFieldIndex(dataField.name()));
+                });
+        for (Map.Entry<String, Map<String, Options>> entry : fileIndexes.entrySet()) {
+            String columnName = entry.getKey();
+            DataField field = map.get(columnName);
+            if (field == null) {
+                throw new IllegalArgumentException(columnName + " does not exist in column fields");
+            }
+            for (Map.Entry<String, Options> typeEntry : entry.getValue().entrySet()) {
+                String indexType = typeEntry.getKey();
+                indexMaintainers.add(
+                        new IndexMaintainer(
+                                columnName,
+                                indexType,
+                                FileIndexer.create(indexType, field.type(), typeEntry.getValue())
+                                        .createWriter(),
+                                InternalRow.createFieldGetter(
+                                        field.type(), index.get(columnName))));
+            }
+        }
+    }
+
+    public void write(InternalRow row) {
+        indexMaintainers.forEach(indexMaintainer -> indexMaintainer.write(row));
+    }
+
+    public void close() throws IOException {
+        if (indexMaintainers.isEmpty()) {

Review Comment:
   should not empty



##########
paimon-core/src/main/java/org/apache/paimon/operation/AppendOnlyFileStoreRead.java:
##########
@@ -162,16 +172,45 @@ public RecordReader<InternalRow> createReader(DataSplit split) throws IOExceptio
                                         Projection.of(dataProjection)
                                                 .project(dataSchema.logicalRowType());
 
-                                return new BulkFormatMapping(
-                                        indexCastMapping.getIndexMapping(),
-                                        indexCastMapping.getCastMapping(),
-                                        partitionPair,
-                                        formatDiscover
-                                                .discover(formatIdentifier)
-                                                .createReaderFactory(
-                                                        projectedRowType, dataFilters));
+                                return Triple.of(
+                                        dataSchema,
+                                        dataFilters,
+                                        new BulkFormatMapping(
+                                                indexCastMapping.getIndexMapping(),
+                                                indexCastMapping.getCastMapping(),
+                                                partitionPair,
+                                                formatDiscover
+                                                        .discover(formatIdentifier)
+                                                        .createReaderFactory(
+                                                                projectedRowType, dataFilters)));
                             });
 
+            TableSchema dataSchema = formatMappingTriple.f0;
+            List<Predicate> dataFilter = formatMappingTriple.f1;
+            BulkFormatMapping bulkFormatMapping = formatMappingTriple.f2;
+            if (dataFilter != null && !dataFilter.isEmpty()) {
+                List<String> indexFiles =
+                        file.extraFiles().stream()
+                                .filter(
+                                        name ->
+                                                name.startsWith(
+                                                        DataFilePathFactory.INDEX_PATH_PREFIX))
+                                .collect(Collectors.toList());
+                if (fileIndexReadEnabled && !indexFiles.isEmpty()) {

Review Comment:
   Here should just be a simple wrapper `RecordReader`. Just like `ApplyDeletionVectorReader`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3177:
URL: https://github.com/apache/paimon/pull/3177#discussion_r1565386638


##########
paimon-core/src/main/java/org/apache/paimon/operation/AppendOnlyFileStoreRead.java:
##########
@@ -66,7 +68,9 @@ public class AppendOnlyFileStoreRead implements FileStoreRead<InternalRow> {
     private final TableSchema schema;
     private final FileFormatDiscover formatDiscover;
     private final FileStorePathFactory pathFactory;
-    private final Map<FormatKey, BulkFormatMapping> bulkFormatMappings;
+    private final Map<FormatKey, Triple<TableSchema, List<Predicate>, BulkFormatMapping>>

Review Comment:
   Maybe it is better to add `TableSchema, List<Predicate>` to `BulkFormatMapping`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3177:
URL: https://github.com/apache/paimon/pull/3177#discussion_r1565375997


##########
paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java:
##########
@@ -78,6 +79,8 @@ public class AppendOnlyWriter implements RecordWriter<InternalRow>, MemoryOwner
     private SinkWriter sinkWriter;
     private final FieldStatsCollector.Factory[] statsCollectors;
     private final IOManager ioManager;
+    private final FileIndexOptions fileIndexes;
+    private final long inManifestThreshold;

Review Comment:
   Put this into `FileIndexOptions`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3177:
URL: https://github.com/apache/paimon/pull/3177#discussion_r1565384310


##########
paimon-core/src/main/java/org/apache/paimon/io/FileIndexFileReader.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.io;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.fileindex.FileIndexPredicate;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.mergetree.compact.ConcatRecordReader;
+import org.apache.paimon.predicate.Predicate;
+import org.apache.paimon.predicate.PredicateBuilder;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.schema.TableSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** File index reader, do the filter in the constructor. */
+public class FileIndexFileReader implements RecordReader<InternalRow> {
+
+    private final RecordReader<InternalRow> reader;
+
+    public FileIndexFileReader(
+            FileIO fileIO,
+            TableSchema dataSchema,
+            List<Predicate> dataFilter,
+            DataFilePathFactory dataFilePathFactory,
+            DataFileMeta file,
+            ConcatRecordReader.ReaderSupplier<InternalRow> readerSupplier)
+            throws IOException {
+        boolean filterThisFile = false;
+        if (dataFilter != null && !dataFilter.isEmpty()) {
+            List<String> indexFiles =
+                    file.extraFiles().stream()
+                            .filter(name -> name.endsWith(DataFilePathFactory.INDEX_PATH_SUFFIX))
+                            .collect(Collectors.toList());
+            if (!indexFiles.isEmpty()) {

Review Comment:
   check size? If size bigger than 1? Throw exception?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce file index read/write framework. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #3177:
URL: https://github.com/apache/paimon/pull/3177


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org