Re: Review Request 68547: SENTRY-2370: Create a cache of PathUpdates to send to NN

[Arjun Mishra via Review Board <no...@reviews.apache.org>]

> On Sept. 28, 2018, 4:56 p.m., Sergio Pena wrote:
> > What if NN does not request full images anymore? How can the cache be released from memory to avoid unused cache for the rest of the process life?
> 
> Arjun Mishra wrote:
>     Sergio, the cache is releases as soon as Deltas are being sent to NN
> 
> Sergio Pena wrote:
>     what if there are no Deltas available to send in the next NN requests? The cache will live on memory until then.
>     Look for Java soft or weak references (and the use of SoftReferences or WeakReferences). These objects might help on letting the GC to clean your cache if it is not needed anymore.

Deltas don't have to be sent they only have to be requested by NN. As soon as NN gets a full update, it will start requesting delta updates and just the trigger for requesting delta updates will invalidate cache
I will research soft references


- Arjun


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68547/#review209098
-----------------------------------------------------------


On Sept. 28, 2018, 7:56 p.m., Arjun Mishra wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68547/
> -----------------------------------------------------------
> 
> (Updated Sept. 28, 2018, 7:56 p.m.)
> 
> 
> Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena.
> 
> 
> Bugs: SENTRY-2370
>     https://issues.apache.org/jira/browse/SENTRY-2370
> 
> 
> Repository: sentry
> 
> 
> Description
> -------
> 
> When NN requests path updates from sentry and if it exceeds the time threshold, on consecutive attempts sentry will attempt to fetch the full update from scratch. Instead it should cache it and update the cache before sending it to NN
> 
> 
> Diffs
> -----
> 
>   sentry-hdfs/sentry-hdfs-common/src/main/java/org/apache/sentry/hdfs/ImageCache.java PRE-CREATION 
>   sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/DBUpdateForwarder.java 08b16a4df 
>   sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/PathsUpdateImageCache.java PRE-CREATION 
>   sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/PermissionsUpdateImageCache.java PRE-CREATION 
>   sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/SentryPlugin.java b8f5ce7db 
>   sentry-hdfs/sentry-hdfs-service/src/test/java/org/apache/sentry/hdfs/TestDBUpdateForwarder.java f86ce6f83 
> 
> 
> Diff: https://reviews.apache.org/r/68547/diff/7/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Arjun Mishra
> 
>