You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "ulysses-you (via GitHub)" <gi...@apache.org> on 2024/09/19 07:44:04 UTC

[PR] [spark] PaimonSplitScan supports column pruning and filter push down [paimon]

ulysses-you opened a new pull request, #4217:
URL: https://github.com/apache/paimon/pull/4217

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   
   `PaimonSplitScan` is built for internal scan with update/delete/mergeinto. It is used to generate deletion vector, collect touched files, etc. The main usage is to select some metadata columns based on target table, e.g., row index, file path. That says, it does not need to load data columns.
   
   This pr makes `PaimonSplitScan` support column pruning and filter push down to improve performance:
   1. introduce `KnownSplitsTable`, it is a `ReadonlyTable` and hold some known data splits
   2. introduce `PaimonSplitScanBuilder`, it is used when the table is the `KnownSplitsTable` and build `PaimonSplitScan`
   
   For example:
   ```
   update test set c1 = 9 where c2 = 'a';
   ```
   
   before:
   ```
   (1) BatchScan default.test
   Output [5]: [c1#197, c2#198, c3#199, c4#200, __paimon_file_path#205]
   class org.apache.paimon.spark.PaimonSplitScan
   
   (2) Filter [codegen id : 1]
   Input [5]: [c1#197, c2#198, c3#199, c4#200, __paimon_file_path#205]
   Condition : (c2#198 = a)
   
   (3) Project [codegen id : 1]
   Output [1]: [__paimon_file_path#205]
   Input [5]: [c1#197, c2#198, c3#199, c4#200, __paimon_file_path#205]
   ```
   
   after:
   
   ```
   (1) BatchScan test-knownsplits
   Output [2]: [c2#137, __paimon_file_path#144]
   PaimonSplitScan: [test], PushedFilters: [Equal(c2, a)]
   
   (2) Filter [codegen id : 1]
   Input [2]: [c2#137, __paimon_file_path#144]
   Condition : (c2#137 = a)
   
   (3) Project [codegen id : 1]
   Output [1]: [__paimon_file_path#144]
   Input [2]: [c2#137, __paimon_file_path#144]
   ```
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   Pass CI
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   No
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [spark] PaimonSplitScan supports column pruning and filter push down [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4217:
URL: https://github.com/apache/paimon/pull/4217#issuecomment-2370920110

   @ulysses-you Can you add a test to verify plan?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [spark] PaimonSplitScan supports column pruning and filter push down [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4217:
URL: https://github.com/apache/paimon/pull/4217


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [spark] PaimonSplitScan supports column pruning and filter push down [paimon]

Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #4217:
URL: https://github.com/apache/paimon/pull/4217#issuecomment-2360356539

   cc @JingsongLi @YannByron thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org