[GitHub] [incubator-druid] sanastas commented on issue #7838: Improve IncrementalIndex concurrency scalability

[GitBox <gi...@apache.org>]

sanastas commented on issue #7838: Improve IncrementalIndex concurrency scalability
URL: https://github.com/apache/incubator-druid/pull/7838#issuecomment-502610905
 
 
   @jihoonson , indeed thank you for raising this important question! I would like to add a bit to @eranmeir answer.
   
   Without relevance to Oak, it might be interesting that other databases found it beneficial to make the inserts to in-memory component concurrent. Here is a [link](http://webee.technion.ac.il/~idish/ftp/clsm.pdf) to academic paper about cLSM (concurrent LSM) showing benefits from making in-memory component concurrent. And here are RocksDB [release notes](https://rocksdb.org/blog/2017/01/06/rocksdb-5-0-1-released.html) from 2017 presenting concurrent writes to memtable enabled as a default: 
   > Options::allow_concurrent_memtable_write and Options::enable_write_thread_adaptive_yield are now true by default.
   
   And with relevance to Oak, OakMap brings with it two benefits: (1) taking the key-value mapping off-heap, (2) high multi-threading scalability. Due to off-heaping, the Oak-based IncrementalIndex might manage more data in-memory thus require less often flushes to disk. Such IncrementalIndex might be responsible for a bigger keys partition and that way cause *less merges*. Clearly, managing bigger map is more efficient when multi-thread writes are allowed and are efficient.  
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org