[GitHub] [incubator-paimon] JingsongLi opened a new pull request, #1975: [core] Introduce index-ttl and bootstrap-min-partition for cross-partition-upsert

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi opened a new pull request, #1975:
URL: https://github.com/apache/incubator-paimon/pull/1975

   
   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Cross-partition-upsert Performance: For tables with a large amount of data, there will be a significant loss in performance. Moreover,
   initialization takes a long time.
   
   If your upsert does not rely on too old data, you can consider configuring index TTL and bootstrap-min-partition to
   reduce Index and initialization time:
   - `'cross-partition-upsert.index-ttl'`: The TTL in rocksdb index, this can avoid maintaining too many indexes and lead
     to worse and worse performance.
   - `'cross-partition-upsert.bootstrap-min-partition'`: The min partition bootstrap of rocksdb index, bootstrap will only
     read the partitions above it, and the smaller partitions will not be read into the index. This can reduce job startup
     time and excessive initialization of index.
   
   But please note that this may also cause data duplication.
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-paimon] JingsongLi merged pull request #1975: [core] Introduce index-ttl and bootstrap-min-partition for cross-partition-upsert

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #1975:
URL: https://github.com/apache/incubator-paimon/pull/1975


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org