You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Aiden-Dong (via GitHub)" <gi...@apache.org> on 2024/11/29 04:27:33 UTC
[PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Aiden-Dong opened a new pull request, #4608:
URL: https://github.com/apache/paimon/pull/4608
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: #4586
优化了基于 Parquet 文件过滤读取时的谓词下推能力，将原先的Parquet 谓词下推由RowGroup级别增强到了 Column page 级别，查询性能提升明显。
Optimized the predicate pushdown capability for filtering and reading Parquet files, enhancing the original predicate pushdown from the RowGroup level to the Column Page level, resulting in a significant improvement in query performance.
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2510673838
Spark Issue: https://issues.apache.org/jira/browse/SPARK-34859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2510825719
@JingsongLi
Based on the error log, it seems that this error is not likely caused by the recent changes.
从错误日志来看，感觉这个错误不太像是改动带来的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2514051926
> Thanks for the testing and PR. I downloaded and tested it immediately. Compared with the previous version, this PR improvement increased the speed of Parquet by nearly 10 times, which is a huge improvement! I compared the orc implementation and considered the issue of testing jit. My test results show that the current Parquet implementation is at least 1 times slower than orc. 感谢做的测试和pr。我第一时间下载了并且测试，相对之前的版本这个pr提升把Parquet的速度提升了接近10倍，非常大的提升！ 我对比了orc实现，考虑了测试jit的问题。 我测试结果现在的Parquet实现比orc还慢至少1倍。
>
> On my computer, the Parquet result is 8.4s and the ORC result is 4.1s. 在我电脑上Parquet结果是8.4s，ORC结果是4.1s。
>
> @Aiden-Dong @JingsongLi @leaves12138
>
> ```java
> Table table = TableUtil.getTable(); // PrimaryKeyFileStoreTable
> PredicateBuilder builder = new PredicateBuilder(
> RowType.of(DataTypes.INT(),
> DataTypes.STRING(),
> DataTypes.STRING()));
>
> int[] projection = new int[] {0, 1, 2};
>
> ReadBuilder readBuilder = table.newReadBuilder()
> .withProjection(projection);
>
> Random random = new Random();
>
> for(int i = 0 ; i < 30 ; i ++){
> InnerTableRead read = (InnerTableRead)readBuilder.newRead();
> int key = random.nextInt(4000000);
>
> Predicate keyFilter = builder.equal(0, key);
>
> InnerTableScan tableScan = (InnerTableScan) readBuilder
> .withFilter(keyFilter)
> .newScan();
> InnerTableScan innerTableScan = tableScan.withFilter(keyFilter);
> TableScan.Plan plan = innerTableScan.plan();
> List<Split> splits = plan.splits();
>
> read.withFilter(keyFilter);//.executeFilter();
> RecordReader<InternalRow> reader = read.createReader(splits);
>
> reader.forEachRemaining(internalRow -> {
>
> int f0 = internalRow.getInt(0);
> String f1 = internalRow.getString(1).toString();
> String f2 = internalRow.getString(2).toString();
> System.out.println(String.format("%d - {%d, %s, %s}",key, f0, f1, f2));
> });
> }
> long startTime = System.currentTimeMillis();
> for(int i = 0 ; i < 1000 ; i ++){
> InnerTableRead read = (InnerTableRead)readBuilder.newRead();
> int key = random.nextInt(4000000);
>
> Predicate keyFilter = builder.equal(0, key);
>
> InnerTableScan tableScan = (InnerTableScan) readBuilder
> .withFilter(keyFilter)
> .newScan();
> InnerTableScan innerTableScan = tableScan.withFilter(keyFilter);
> TableScan.Plan plan = innerTableScan.plan();
> List<Split> splits = plan.splits();
>
> read.withFilter(keyFilter);//.executeFilter();
> RecordReader<InternalRow> reader = read.createReader(splits);
>
> reader.forEachRemaining(internalRow -> {
>
> int f0 = internalRow.getInt(0);
> String f1 = internalRow.getString(1).toString();
> String f2 = internalRow.getString(2).toString();
> System.out.println(String.format("%d - {%d, %s, %s}",key, f0, f1, f2));
> });
> }
> long stopTime = System.currentTimeMillis();
> System.out.println("time : " + (stopTime - startTime));
> ```
>
> writer see #4586
Thanks @ranxianglei for testing, I will follow up this question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #4608:
URL: https://github.com/apache/paimon/pull/4608#discussion_r1870516676
##########
paimon-format/src/main/java/org/apache/paimon/format/parquet/reader/ShortColumnReader.java:
##########
@@ -71,6 +71,38 @@ protected void readBatch(int rowId, int num, WritableShortVector column) {
}
}
+ @Override
+ protected void skipBatch(int num) {
+ int left = num;
+ while (left > 0) {
+ if (runLenDecoder.currentCount == 0) {
+ runLenDecoder.readNextGroup();
+ }
+ int n = Math.min(left, runLenDecoder.currentCount);
+ switch (runLenDecoder.mode) {
+ case RLE:
+ if (runLenDecoder.currentValue == maxDefLevel) {
+ skipShot(n);
Review Comment:
maybe skipShort?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on code in PR #4608:
URL: https://github.com/apache/paimon/pull/4608#discussion_r1866945845
##########
paimon-core/src/test/java/org/apache/paimon/table/PrimaryKeyFileStoreTableTest.java:
##########
@@ -810,6 +810,65 @@ public void testDeletionVectorsWithFileIndexInFile() throws Exception {
"1|4|500|binary|varbinary|mapKey:mapVal|multiset"));
}
+ @Test
+ public void testDeletionVectorsWithParquetFilter() throws Exception {
+ FileStoreTable table =
+ createFileStoreTable(
+ conf -> {
+ conf.set(BUCKET, 1);
+ conf.set(DELETION_VECTORS_ENABLED, true);
+ conf.set(FILE_FORMAT, "parquet");
+ conf.set("parquet.block.size", "1048576");
+ conf.set("parquet.page.size", "1024");
+ });
+
+ BatchWriteBuilder writeBuilder = table.newBatchWriteBuilder();
+
+ BatchTableWrite write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+
+ for (int i = 0; i < 2000; i++) {
+ write.write(rowData(i, i, i * 100L));
+ }
+
+ List<CommitMessage> messages = write.prepareCommit();
+ BatchTableCommit commit = writeBuilder.newCommit();
+ commit.commit(messages);
+ write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+ for (int i = 1000; i < 2000; i++) {
+ write.write(rowDataWithKind(RowKind.DELETE, i, i, i * 100L));
+ }
+
+ messages = write.prepareCommit();
+ commit = writeBuilder.newCommit();
+ commit.commit(messages);
+
+ PredicateBuilder builder = new PredicateBuilder(ROW_TYPE);
+ List<Split> splits = toSplits(table.newSnapshotReader().read().dataSplits());
+
+ for (int i = 500; i < 510; i++) {
+ TableRead read = table.newRead().withFilter(builder.equal(0, i)).executeFilter();
+ assertThat(getResult(read, splits, BATCH_ROW_TO_STRING))
+ .isEqualTo(
+ Arrays.asList(
+ String.format(
+ "%d|%d|%d|binary|varbinary|mapKey:mapVal|multiset",
+ i, i, i * 100L)));
+ }
+
+ for (int i = 1500; i < 1510; i++) {
+ TableRead read = table.newRead().withFilter(builder.equal(0, i)).executeFilter();
Review Comment:
I misunderstood the first column as the primary key... I'll make the correction.
我把第一列理解成了主键。。我改一改
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2516426450
This method return the nextIndex seems not correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2516494648
I've submitted a PR to fix this logic ： https://github.com/apache/paimon/pull/4636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #4608:
URL: https://github.com/apache/paimon/pull/4608#discussion_r1866941728
##########
paimon-core/src/test/java/org/apache/paimon/table/PrimaryKeyFileStoreTableTest.java:
##########
@@ -810,6 +810,65 @@ public void testDeletionVectorsWithFileIndexInFile() throws Exception {
"1|4|500|binary|varbinary|mapKey:mapVal|multiset"));
}
+ @Test
+ public void testDeletionVectorsWithParquetFilter() throws Exception {
+ FileStoreTable table =
+ createFileStoreTable(
+ conf -> {
+ conf.set(BUCKET, 1);
+ conf.set(DELETION_VECTORS_ENABLED, true);
+ conf.set(FILE_FORMAT, "parquet");
+ conf.set("parquet.block.size", "1048576");
+ conf.set("parquet.page.size", "1024");
+ });
+
+ BatchWriteBuilder writeBuilder = table.newBatchWriteBuilder();
+
+ BatchTableWrite write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+
+ for (int i = 0; i < 2000; i++) {
+ write.write(rowData(i, i, i * 100L));
Review Comment:
If you want test datas in one bucket in deletion vector, just rowData(0, i, i*100L), the first column is "pt" for partition .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #4608:
URL: https://github.com/apache/paimon/pull/4608#discussion_r1866942913
##########
paimon-core/src/test/java/org/apache/paimon/table/PrimaryKeyFileStoreTableTest.java:
##########
@@ -810,6 +810,65 @@ public void testDeletionVectorsWithFileIndexInFile() throws Exception {
"1|4|500|binary|varbinary|mapKey:mapVal|multiset"));
}
+ @Test
+ public void testDeletionVectorsWithParquetFilter() throws Exception {
+ FileStoreTable table =
+ createFileStoreTable(
+ conf -> {
+ conf.set(BUCKET, 1);
+ conf.set(DELETION_VECTORS_ENABLED, true);
+ conf.set(FILE_FORMAT, "parquet");
+ conf.set("parquet.block.size", "1048576");
+ conf.set("parquet.page.size", "1024");
+ });
+
+ BatchWriteBuilder writeBuilder = table.newBatchWriteBuilder();
+
+ BatchTableWrite write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+
+ for (int i = 0; i < 2000; i++) {
+ write.write(rowData(i, i, i * 100L));
+ }
+
+ List<CommitMessage> messages = write.prepareCommit();
+ BatchTableCommit commit = writeBuilder.newCommit();
+ commit.commit(messages);
+ write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+ for (int i = 1000; i < 2000; i++) {
+ write.write(rowDataWithKind(RowKind.DELETE, i, i, i * 100L));
+ }
+
+ messages = write.prepareCommit();
+ commit = writeBuilder.newCommit();
+ commit.commit(messages);
+
+ PredicateBuilder builder = new PredicateBuilder(ROW_TYPE);
+ List<Split> splits = toSplits(table.newSnapshotReader().read().dataSplits());
+
+ for (int i = 500; i < 510; i++) {
+ TableRead read = table.newRead().withFilter(builder.equal(0, i)).executeFilter();
Review Comment:
maybe builder.equal(1, i) is what you want. Partition predicate will not push down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 merged PR #4608:
URL: https://github.com/apache/paimon/pull/4608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2516421696
@Aiden-Dong Hi, I still have some questions about the pull request. How can I get touch with you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #4608:
URL: https://github.com/apache/paimon/pull/4608#discussion_r1866942339
##########
paimon-core/src/test/java/org/apache/paimon/table/PrimaryKeyFileStoreTableTest.java:
##########
@@ -810,6 +810,65 @@ public void testDeletionVectorsWithFileIndexInFile() throws Exception {
"1|4|500|binary|varbinary|mapKey:mapVal|multiset"));
}
+ @Test
+ public void testDeletionVectorsWithParquetFilter() throws Exception {
+ FileStoreTable table =
+ createFileStoreTable(
+ conf -> {
+ conf.set(BUCKET, 1);
+ conf.set(DELETION_VECTORS_ENABLED, true);
+ conf.set(FILE_FORMAT, "parquet");
+ conf.set("parquet.block.size", "1048576");
+ conf.set("parquet.page.size", "1024");
+ });
+
+ BatchWriteBuilder writeBuilder = table.newBatchWriteBuilder();
+
+ BatchTableWrite write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+
+ for (int i = 0; i < 2000; i++) {
+ write.write(rowData(i, i, i * 100L));
+ }
+
+ List<CommitMessage> messages = write.prepareCommit();
+ BatchTableCommit commit = writeBuilder.newCommit();
+ commit.commit(messages);
+ write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+ for (int i = 1000; i < 2000; i++) {
+ write.write(rowDataWithKind(RowKind.DELETE, i, i, i * 100L));
Review Comment:
same rowDataWithKind(RowKind.DELETE, 0, i, i * 100L)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #4608:
URL: https://github.com/apache/paimon/pull/4608#discussion_r1866944468
##########
paimon-core/src/test/java/org/apache/paimon/table/PrimaryKeyFileStoreTableTest.java:
##########
@@ -810,6 +810,65 @@ public void testDeletionVectorsWithFileIndexInFile() throws Exception {
"1|4|500|binary|varbinary|mapKey:mapVal|multiset"));
}
+ @Test
+ public void testDeletionVectorsWithParquetFilter() throws Exception {
+ FileStoreTable table =
+ createFileStoreTable(
+ conf -> {
+ conf.set(BUCKET, 1);
+ conf.set(DELETION_VECTORS_ENABLED, true);
+ conf.set(FILE_FORMAT, "parquet");
+ conf.set("parquet.block.size", "1048576");
+ conf.set("parquet.page.size", "1024");
+ });
+
+ BatchWriteBuilder writeBuilder = table.newBatchWriteBuilder();
+
+ BatchTableWrite write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+
+ for (int i = 0; i < 2000; i++) {
+ write.write(rowData(i, i, i * 100L));
Review Comment:
for (int i = 0; i < 2000; i++) {
write.write(rowData(i, i, i * 100L));
The code above will write 2000 records in 2000 partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2513463493
I have revised the unit tests and added the write cardinality to ensure that there are more pages and RowGroups in the Parquet file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2510864648
> @JingsongLi
>
> Based on the error log, it seems that this error is not likely caused by the recent changes. 从错误日志来看，感觉这个错误不太像是改动带来的。
You can enable action in your repo to double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2510824710
@JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "ranxianglei (via GitHub)" <gi...@apache.org>.
ranxianglei commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2514015603
Thanks for the testing and PR. I downloaded and tested it immediately. Compared with the previous version, this PR improvement increased the speed of Parquet by nearly 10 times, which is a huge improvement!
I compared the orc implementation and considered the issue of testing jit.
My test results show that the current Parquet implementation is at least 1 times slower than orc.
感谢做的测试和pr。我第一时间下载了并且测试，相对之前的版本这个pr提升把Parquet的速度提升了接近10倍，非常大的提升！
我对比了orc实现，考虑了测试jit的问题。
我测试结果现在的Parquet实现比orc还慢至少1倍。
On my computer, the Parquet result is 8.4s and the ORC result is 4.1s.
在我电脑上Parquet结果是8.4s，ORC结果是4.1s。
@Aiden-Dong @JingsongLi @leaves12138
```java
Table table = TableUtil.getTable(); // PrimaryKeyFileStoreTable
PredicateBuilder builder = new PredicateBuilder(
RowType.of(DataTypes.INT(),
DataTypes.STRING(),
DataTypes.STRING()));
int[] projection = new int[] {0, 1, 2};
ReadBuilder readBuilder = table.newReadBuilder()
.withProjection(projection);
Random random = new Random();
for(int i = 0 ; i < 30 ; i ++){
InnerTableRead read = (InnerTableRead)readBuilder.newRead();
int key = random.nextInt(4000000);
Predicate keyFilter = builder.equal(0, key);
InnerTableScan tableScan = (InnerTableScan) readBuilder
.withFilter(keyFilter)
.newScan();
InnerTableScan innerTableScan = tableScan.withFilter(keyFilter);
TableScan.Plan plan = innerTableScan.plan();
List<Split> splits = plan.splits();
read.withFilter(keyFilter);//.executeFilter();
RecordReader<InternalRow> reader = read.createReader(splits);
reader.forEachRemaining(internalRow -> {
int f0 = internalRow.getInt(0);
String f1 = internalRow.getString(1).toString();
String f2 = internalRow.getString(2).toString();
System.out.println(String.format("%d - {%d, %s, %s}",key, f0, f1, f2));
});
}
long startTime = System.currentTimeMillis();
for(int i = 0 ; i < 1000 ; i ++){
InnerTableRead read = (InnerTableRead)readBuilder.newRead();
int key = random.nextInt(4000000);
Predicate keyFilter = builder.equal(0, key);
InnerTableScan tableScan = (InnerTableScan) readBuilder
.withFilter(keyFilter)
.newScan();
InnerTableScan innerTableScan = tableScan.withFilter(keyFilter);
TableScan.Plan plan = innerTableScan.plan();
List<Split> splits = plan.splits();
read.withFilter(keyFilter);//.executeFilter();
RecordReader<InternalRow> reader = read.createReader(splits);
reader.forEachRemaining(internalRow -> {
int f0 = internalRow.getInt(0);
String f1 = internalRow.getString(1).toString();
String f2 = internalRow.getString(2).toString();
System.out.println(String.format("%d - {%d, %s, %s}",key, f0, f1, f2));
});
}
long stopTime = System.currentTimeMillis();
System.out.println("time : " + (stopTime - startTime));
```
writer see https://github.com/apache/paimon/issues/4586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2516294765
> Thanks @ranxianglei for testing, I will follow up this question.
@leaves12138 @ranxianglei
I found that the default stripe size and row index stride of ORC are twice as large as those of Parquet, which leads to a finer granularity of indexes in ORC.
我发现 ORC 默认的 stripe 大小以及 row index stride 大小是 parquet 默认配置的两倍， 这导致 ORC 的索引粒度更小
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2510677628
> @Aiden-Dong Can you add test for parquet page predicate pushdown and deletion vectors enabled? I just want to make sure `currentRowPosition` in `ParquetReaderFactory` still works good.
确实这个 currentRowPosition 在 deletion vector 启用时索引存在问题， 等我我修复一下这个功能，并加入测试
Indeed, the currentRowPosition has an indexing issue when the deletion vector is enabled. Let me fix this functionality and add tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2509731545
@Aiden-Dong Can you add test for parquet page predicate pushdown and deletion vectors enabled? I just want to make sure `currentRowPosition` in `ParquetReaderFactory` still works good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2507118994
Looks very nice! Thanks @Aiden-Dong , I will take a review next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2516434848
> @Aiden-Dong 嗨，我对拉取请求还有一些疑问。我怎样才能联系到你们？
稍等我看一下这个逻辑。
我的微信 : aiden_dong_private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2516428110
It just return the rowIndex in current row group. But what we want is the exactly row index within the whole parquet file. Like:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4608:
URL: https://github.com/apache/paimon/pull/4608#issuecomment-2507588936
我用[#4586] 提到测试样例，生成400万测试数据。
30次随机读取，原先需要12s-17s，优化后大概需要1s 左右。
Using the test case mentioned in [#4586], I generated 4 million test records.
With 30 random reads, the original implementation took around 12-17 seconds, while the optimized version reduced this to approximately 1 second.
before
<img width="1916" alt="image" src="https://github.com/user-attachments/assets/e95f2fca-fd97-4312-9a1e-aea970482224">
after
<img width="1910" alt="image" src="https://github.com/user-attachments/assets/f085dacb-37d1-49c6-b131-9c88dc030df2">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimization of Parquet Predicate Pushdown Capability [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #4608:
URL: https://github.com/apache/paimon/pull/4608#discussion_r1866943395
##########
paimon-core/src/test/java/org/apache/paimon/table/PrimaryKeyFileStoreTableTest.java:
##########
@@ -810,6 +810,65 @@ public void testDeletionVectorsWithFileIndexInFile() throws Exception {
"1|4|500|binary|varbinary|mapKey:mapVal|multiset"));
}
+ @Test
+ public void testDeletionVectorsWithParquetFilter() throws Exception {
+ FileStoreTable table =
+ createFileStoreTable(
+ conf -> {
+ conf.set(BUCKET, 1);
+ conf.set(DELETION_VECTORS_ENABLED, true);
+ conf.set(FILE_FORMAT, "parquet");
+ conf.set("parquet.block.size", "1048576");
+ conf.set("parquet.page.size", "1024");
+ });
+
+ BatchWriteBuilder writeBuilder = table.newBatchWriteBuilder();
+
+ BatchTableWrite write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+
+ for (int i = 0; i < 2000; i++) {
+ write.write(rowData(i, i, i * 100L));
+ }
+
+ List<CommitMessage> messages = write.prepareCommit();
+ BatchTableCommit commit = writeBuilder.newCommit();
+ commit.commit(messages);
+ write =
+ (BatchTableWrite)
+ writeBuilder
+ .newWrite()
+ .withIOManager(new IOManagerImpl(tempDir.toString()));
+ for (int i = 1000; i < 2000; i++) {
+ write.write(rowDataWithKind(RowKind.DELETE, i, i, i * 100L));
+ }
+
+ messages = write.prepareCommit();
+ commit = writeBuilder.newCommit();
+ commit.commit(messages);
+
+ PredicateBuilder builder = new PredicateBuilder(ROW_TYPE);
+ List<Split> splits = toSplits(table.newSnapshotReader().read().dataSplits());
+
+ for (int i = 500; i < 510; i++) {
+ TableRead read = table.newRead().withFilter(builder.equal(0, i)).executeFilter();
+ assertThat(getResult(read, splits, BATCH_ROW_TO_STRING))
+ .isEqualTo(
+ Arrays.asList(
+ String.format(
+ "%d|%d|%d|binary|varbinary|mapKey:mapVal|multiset",
+ i, i, i * 100L)));
+ }
+
+ for (int i = 1500; i < 1510; i++) {
+ TableRead read = table.newRead().withFilter(builder.equal(0, i)).executeFilter();
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org