[GitHub] [incubator-druid] swapneshgandhi opened a new pull request #8272: Support incremental load in Druid lookups

[GitBox <gi...@apache.org>]

swapneshgandhi opened a new pull request #8272: Support incremental load in Druid lookups
URL: https://github.com/apache/incubator-druid/pull/8272
 
 
   
   This commit adds support for incremental load in druid lookups. 
   
   ### Description
   
   For larger lookups with millions of rows, reloading of the lookup is very hard on the JVM due to GC pressure. After this change, during reload only entries that were modified after the last db update would be added to the same underlying cache object.
   Funnily enough, Druid docs say that this feature is already supported in Druid but that wasn't the case.
   
   Changes:
   In JdbcCacheGenerator, only read newly modified rows from the Database. 
   in CacheScheduler, added new method createFromExistingCache to add cache entries to the same underlying cache.
   additions to NamespaceExtractionCacheManager interface for creating new cache object from the same underlying ConcurrentMap.
   
   current limitations: 
   1. incremental load is not supported for OffHeap lookups
   2. Does not handle the deletes from the DB.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
      - [x] using the [concurrency checklist](https://github.com/apache/incubator-druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [x] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths.
   - [ ] added integration tests.
   - [x] been tested in a test Druid cluster.
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `JdbcCacheGenerator`
    * `CacheScheduler`
    * `NamespaceExtractionCacheManager`
    * `OnHeapNamespaceExtractionCacheManager`
    * `OffHeapNamespaceExtractionCacheManager`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org