[GitHub] [pulsar] merlimat commented on issue #4158: Create more offloader indices per each block uploaded to 2nd tier

[GitBox <gi...@apache.org>]

merlimat commented on issue #4158: Create more offloader indices per each block uploaded to 2nd tier
URL: https://github.com/apache/pulsar/pull/4158#issuecomment-487661431
 
 
   > In my opinion, the issue here is really that we are calling seek each time in the ReadHandle implementation. 
   
   I didn't really want to get much in the implementation, just apply the easiest fix. All considered, I think the index overhead is still negligible compared to the data size.
   
   > What are the before and after numbers for this?
   
   
   Before: 
   
   ```
   presto> SELECT COUNT(*) FROM pulsar."public/default".tweets;
    _col0
   --------
    255251
   (1 row)
   
   Query 20190427_062727_00031_cihap, FINISHED, 4 nodes
   Splits: 33 total, 33 done (100.00%)
   1:48 [255K rows, 68.4MB] [2.35K rows/s, 646KB/s]
   ```
   
   After: 
   
   ```
   presto> SELECT COUNT(*) FROM pulsar."public/default".tweets;
     _col0
   ---------
    7810072
   (1 row)
   
   Query 20190429_165451_00016_cihap, FINISHED, 4 nodes
   Splits: 33 total, 33 done (100.00%)
   0:13 [7.81M rows, 2.22GB] [598K rows/s, 174MB/s]
   ```
   
   > What is the benchmark setup?
   
   4 nodes reading from S3
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services