You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "hang8929201 (via GitHub)" <gi...@apache.org> on 2024/11/07 07:07:49 UTC
[PR] [common][core][format] Support using file index result for RowGroup filtering. [paimon]
hang8929201 opened a new pull request, #4473:
URL: https://github.com/apache/paimon/pull/4473
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
<!-- What is the purpose of the change -->
Use file index result(bitmap) for more fine-grained IO clipping in the reader.
### Tests
This is a query in our real business scenario.
- no index:
- index for parquet block filtering:
- index for orc rowgroup filtering:
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core][format] Support using file index result for RowGroup filtering. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4473:
URL: https://github.com/apache/paimon/pull/4473#discussion_r1834067900
##########
paimon-common/src/main/java/org/apache/paimon/fileindex/FileIndexPredicate.java:
##########
@@ -67,6 +67,16 @@ public FileIndexPredicate(SeekableInputStream inputStream, RowType fileRowType)
this.reader = FileIndexFormat.createReader(inputStream, fileRowType);
}
+ public FileIndexResult evaluate(@Nullable Predicate predicate) {
+ if (predicate == null) {
+ return REMAIN;
+ }
+ Set<String> requiredFieldNames = getRequiredNames(predicate);
+ Map<String, Collection<FileIndexReader>> indexReaders = new HashMap<>();
+ requiredFieldNames.forEach(name -> indexReaders.put(name, reader.readColumnIndex(name)));
+ return new FileIndexPredicateTest(indexReaders).test(predicate);
+ }
+
public boolean testPredicate(@Nullable Predicate filePredicate) {
Review Comment:
remove this method? just use `evaluate`.
##########
paimon-common/src/main/java/org/apache/paimon/format/FormatReaderContext.java:
##########
@@ -28,13 +29,18 @@ public class FormatReaderContext implements FormatReaderFactory.Context {
private final FileIO fileIO;
private final Path file;
private final long fileSize;
+ private FileIndexResult fileIndexResult;
Review Comment:
add `@Nullable`
##########
paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetUtil.java:
##########
@@ -79,7 +79,9 @@ public class ParquetUtil {
*/
public static ParquetFileReader getParquetReader(FileIO fileIO, Path path) throws IOException {
Review Comment:
add fileIndex to arguements. this method looks a generic method, should have index.
##########
paimon-common/src/main/java/org/apache/paimon/format/FormatReaderContext.java:
##########
@@ -28,13 +29,18 @@ public class FormatReaderContext implements FormatReaderFactory.Context {
private final FileIO fileIO;
private final Path file;
private final long fileSize;
+ private FileIndexResult fileIndexResult;
public FormatReaderContext(FileIO fileIO, Path file, long fileSize) {
this.fileIO = fileIO;
this.file = file;
this.fileSize = fileSize;
}
+ public void setFileIndexResult(FileIndexResult fileIndexResult) {
Review Comment:
Add a new constructor `FormatReaderContext(FileIO fileIO, Path file, long fileSize, @Nullable FileIndexResult fileIndexResult)`
##########
paimon-core/src/main/java/org/apache/paimon/operation/RawFileSplitRead.java:
##########
@@ -191,26 +192,28 @@ private RecordReader<InternalRow> createFileReader(
BulkFormatMapping bulkFormatMapping,
IOExceptionSupplier<DeletionVector> dvFactory)
throws IOException {
+ FileIndexResult fileIndexResult = null;
if (fileIndexReadEnabled) {
- boolean skip =
- FileIndexSkipper.skip(
Review Comment:
Can you remove `FileIndexSkipper` file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core][format] Support using file index result for RowGroup filtering. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4473:
URL: https://github.com/apache/paimon/pull/4473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core][format] Support using file index result for RowGroup filtering. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4473:
URL: https://github.com/apache/paimon/pull/4473#discussion_r1835908162
##########
paimon-format/src/main/java/org/apache/paimon/format/orc/OrcReaderFactory.java:
##########
@@ -338,6 +344,14 @@ public static org.apache.orc.Reader createReader(
// configure filesystem from Paimon FileIO
readerOptions.filesystem(new HadoopReadOnlyFileSystem(fileIO));
- return OrcFile.createReader(hPath, readerOptions);
+ return new ReaderImpl(hPath, readerOptions) {
+ @Override
+ public RecordReader rows(Options options) throws IOException {
+ return new RecordReaderImpl(this, options, fileIndexResult);
+ }
+ };
}
}
+/*
Review Comment:
???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core][format] Support using file index result for RowGroup filtering. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4473:
URL: https://github.com/apache/paimon/pull/4473#issuecomment-2464432970
Left comments @hang8929201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core][format] Support using file index result for RowGroup filtering. [paimon]
Posted by "hang8929201 (via GitHub)" <gi...@apache.org>.
hang8929201 closed pull request #4473: [common][core][format] Support using file index result for RowGroup filtering.
URL: https://github.com/apache/paimon/pull/4473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core][format] Support using file index result for RowGroup filtering. [paimon]
Posted by "hang8929201 (via GitHub)" <gi...@apache.org>.
hang8929201 commented on code in PR #4473:
URL: https://github.com/apache/paimon/pull/4473#discussion_r1832226440
##########
paimon-core/src/main/java/org/apache/paimon/operation/RawFileSplitRead.java:
##########
@@ -191,26 +192,38 @@ private RecordReader<InternalRow> createFileReader(
BulkFormatMapping bulkFormatMapping,
IOExceptionSupplier<DeletionVector> dvFactory)
throws IOException {
+ FileIndexResult fileIndexResult = null;
if (fileIndexReadEnabled) {
- boolean skip =
- FileIndexSkipper.skip(
+ // boolean skip =
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core][format] Support using file index result for RowGroup filtering. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4473:
URL: https://github.com/apache/paimon/pull/4473#discussion_r1832211430
##########
paimon-core/src/main/java/org/apache/paimon/operation/RawFileSplitRead.java:
##########
@@ -191,26 +192,38 @@ private RecordReader<InternalRow> createFileReader(
BulkFormatMapping bulkFormatMapping,
IOExceptionSupplier<DeletionVector> dvFactory)
throws IOException {
+ FileIndexResult fileIndexResult = null;
if (fileIndexReadEnabled) {
- boolean skip =
- FileIndexSkipper.skip(
+ // boolean skip =
Review Comment:
Maybe you want to remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org