[GitHub] [incubator-pinot] npawar opened a new issue #4801: Smarter resize buffer and algorithm resizing in IndexedTable

[GitBox <gi...@apache.org>]

npawar opened a new issue #4801: Smarter resize buffer and algorithm resizing in IndexedTable
URL: https://github.com/apache/incubator-pinot/issues/4801
 
 
   The resizer has a very naive logic at the moment to decide the buffer capacity:
   From BaseTable.java:
   ```
   // TODO: tune these numbers
         // Based on the capacity and maxCapacity, the resizer will smartly choose to evict/retain recors from the PQ
         if (capacity <= 100_000) { // Capacity is small, make a very large buffer. Make PQ of records to retain, during resize
           _maxCapacity = 1_000_000;
         } else { // Capacity is large, make buffer only slightly bigger. Make PQ of records to evict, during resize
           _maxCapacity = (int) (capacity * 1.2);
         }
   ```
   This allocation is unfair for numbers which fall on the borderline. We need smarter formulas for computing this.
   
   The resize algorithm (in TableResizer) also needs a revisit and enhancement

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org