Errors loading a DirectMemory Map

[Scott Selvia <ss...@datamentors.com>]

I'm using the following code to initialize my map, I'm also trying to
get a handle the amount of memory to allocate.  Using the code below the
memory starts off right around 3 GB as I would expect.  However, it
keeps increasing up to 7 GB.  I'm trying to load a map of unique
addresses with a count of the number of hits for each.  I'm starting
small and using only 5 million of the 270 million records.  I've
performed a dump from the database of all of the unique values and in
CSV form it is only 2.5 GB, hence the 3 GB starting point for the
DirectMemory cache.

 

AllocationPolicy allocationPolicy = new RoundRobinAllocationPolicy();

MemoryManagerService<Integer> memoryManager = new
MemoryManagerServiceImpl<>( allocationPolicy, true );

CacheService<String, Integer> cache = new DirectMemory<String,
Integer>().setMap(new ConcurrentHashMap<String,
Pointer<Integer>>()).setMemoryManager(memoryManager).setNumberOfBuffers(
3).setSize(Ram.Mb(1024)).setInitialCapacity(Ram.Mb(1024)).setConcurrency
Level(4).newCacheService();

 

As I read an load the file, I get about 400K in and I get the following
stacktrace, but the process continues and finally around 1.2 Million
records processed I get the second stacktrace.  I'm using the following
code to retrieve the current count value and put the count back into the
map.  I've changed the code to only process 500K records, when I dump
the map contents I have a Number of Entries equal to NULL.  That part I
don't understand, since I'm setting either a 1 or a count + 1 value.

 

Questions:

 

1.       Am I setting the CacheService correctly to allocate 3 GB of
Memory?

2.       Why is the Memory going up to 7 GB?

3.       Should I be "Getting" or "Putting" the value differently from
or to the DirectMemory Map?

 

Integer count = cache.retrieve(temp);

if (count != null)

{

   cache.put(temp, count + 1);

}

else

{

   cache.put(temp, 1);

}

 

100000

200000

300000

400000

Exception in thread "Timer-0" java.lang.IllegalArgumentException: The
buffer
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$Merg
ingNioMemoryBuffer@25015757 seems not to belong to this allocator

                at
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$Merg
ingNioMemoryBuffer.free(MergingByteBufferAllocator.java:396)

                at
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator.free
(MergingByteBufferAllocator.java:157)

                at
org.apache.directmemory.memory.MemoryManagerServiceImpl.free(MemoryManag
erServiceImpl.java:185)

                at
org.apache.directmemory.memory.AbstractMemoryManager.free(AbstractMemory
Manager.java:149)

                at
org.apache.directmemory.memory.AbstractMemoryManager.collectLFU(Abstract
MemoryManager.java:139)

                at
org.apache.directmemory.cache.CacheServiceImpl.collectLFU(CacheServiceIm
pl.java:260)

                at
org.apache.directmemory.cache.CacheServiceImpl$1.run(CacheServiceImpl.ja
va:85)

                at java.util.TimerThread.mainLoop(Timer.java:555)

                at java.util.TimerThread.run(Timer.java:505)

500000

600000

700000

800000

900000

1000000

1100000

1200000

java.lang.IllegalArgumentException: The buffer
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$Merg
ingNioMemoryBuffer@39e2ab21 seems not to belong to this allocator

                at
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$Merg
ingNioMemoryBuffer.free(MergingByteBufferAllocator.java:396)

                at
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator.free
(MergingByteBufferAllocator.java:157)

                at
org.apache.directmemory.memory.MemoryManagerServiceImpl.free(MemoryManag
erServiceImpl.java:185)

                at
org.apache.directmemory.cache.CacheServiceImpl.store(CacheServiceImpl.ja
va:148)

                at
org.apache.directmemory.cache.CacheServiceImpl.put(CacheServiceImpl.java
:118)

                at
org.apache.directmemory.cache.CacheServiceImpl.put(CacheServiceImpl.java
:109)

                at
com.datamentors.CreateVoltDBLoadFileAndMaps.<init>(CreateMaps.java:150)

                at
com.datamentors.CreateVoltDBLoadFileAndMaps.main(CreateMaps.java:290)

Scott Selvia
(O) 813.960.7800 x102
(F) 813.960.7811
http://www.datamentors.com <http://www.datamentors.com/>