[jira] [Created] (NIFI-3582) DistributedMapCacheServer does not evict persistent records

["Joseph Gresock (JIRA)" <ji...@apache.org>]

Joseph Gresock created NIFI-3582:
------------------------------------

             Summary: DistributedMapCacheServer does not evict persistent records
                 Key: NIFI-3582
                 URL: https://issues.apache.org/jira/browse/NIFI-3582
             Project: Apache NiFi
          Issue Type: Bug
          Components: Core Framework
    Affects Versions: 1.1.1
            Reporter: Joseph Gresock
            Assignee: Mark Payne


The PersistentMapCache does not appear to be evicting records when used by the DistributedMapCacheServer, at least in the FIFO eviction mode.  

To replicate this behavior in NiFi 1.1.1, add the following to your flow:
*Services*
*DistributedMapCacheServer, port 4557, maximum cache entries = 10,000, FIFO eviction, persistence directory specified
*DistributedMapCacheClientService, point to the same host and port

*Flow*
GenerateFlowFile (randomize 1K binary files in batches of 10, schedule 10 threads) -> HashContent (md5) into hash.value -> DetectDuplicate with identifier = $\{hash.value\}, description = ., no age off, select your cache client, cache identifier = true

This should cause the snapshot file in your cache's persistence directory to exceed 100,000 keys pretty quickly, and as far as I can tell, it never goes back down.  

On our production system, we have the cache server configured for 100,000 max entries with FIFO eviction, and we recently saw this log statement showing the writeahead log with over 4 million entries:
{code}
nifi-app.log:2017-03-09 15:03:00,670 INFO [Distributed Cache Server Communications Thread: ac907dec-49a4-439e-99f5-1558f2358d87] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@40569408 
checkpointed with 4262902 Records and 0 Swap Files in 256302 milliseconds (Stop-the-world time = 1378 milliseconds, Clear Edit Logs time = 19 millis), max Transaction ID 4263237
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)