You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@directmemory.apache.org by Scott Selvia <ss...@datamentors.com> on 2014/04/16 21:13:02 UTC
Loading Map hangs/boggs down once Initial Size is Reached
I'm loading a map with String, Integer values. The cacheservice is
initialized with the following code, the code works great until the
amount of memory "Used" reaches the initial size value when the cache is
declared. When the application starts the memory starts at 4 GB and
works it way up, at the point of bogging down the memory used by the
java process is 12 GB. The application is still running but it is very
slowwwwww.
AllocationPolicy allocationPolicy = new RoundRobinAllocationPolicy();
MemoryManagerService<Integer> memoryManager = new
MemoryManagerServiceImpl<>( allocationPolicy, true );
CacheService<String, Integer> cache = new DirectMemory<String,
Integer>().setMap(new ConcurrentSkipListMap<String,
Pointer<Integer>>()).setMemoryManager(memoryManager).setNumberOfBuffers(
4).setSize(Ram.Mb(1024)).setInitialCapacity(Ram.Mb(1024)).setConcurrency
Level(4).setDisposalTime((60 * 60 * 1000) *
3).setSerializer(SerializerFactory.createNewSerializer()).newCacheServic
e();
I've changed the code and simplified the cache creation line and took
out the disposal time, concurrency, and serializer. What I noticed when
I executed the code the amount of USED memory dropped and finally I got
a stacktrace. If items are being removed or rotated from the map, that
would not be good. I'm trying to build a unique list of items and I
don't want key value pairs rolling off the list.
CacheService<String, Integer> cache = new DirectMemory<String,
Integer>().setMap(new ConcurrentSkipListMap<String,
Pointer<Integer>>()).setMemoryManager(memoryManager).setNumberOfBuffers(
4).setSize(1024000000).newCacheService();
What is the best approach using DirectMemory to keep all of my items on
the list? There is not a lot of great information yet regarding best
practices, any help would be appreciated.
Scott Selvia
(O) 813.960.7800 x102
(F) 813.960.7811
http://www.datamentors.com <http://www.datamentors.com/>
RE: Loading Map hangs/boggs down once Initial Size is Reached
Posted by Scott Selvia <ss...@datamentors.com>.
I forgot to add the detail lines and the stacktrace for the change to
the second CacheService statement change. Please note, the used memory
starts to drop at 14,000,000 million records processed. The size of the
map continues to go up but I'm not convinced that the process is not
adding them faster than the cache service is removing them?
1000000 | Size of Map = 408367 | Capacity = 3,906.3Mb | Used = 29.9Mb
2000000 | Size of Map = 810361 | Capacity = 3,906.3Mb | Used = 52.9Mb
3000000 | Size of Map = 1207693 | Capacity = 3,906.3Mb | Used = 70.7Mb
4000000 | Size of Map = 1608967 | Capacity = 3,906.3Mb | Used = 85.2Mb
5000000 | Size of Map = 2012128 | Capacity = 3,906.3Mb | Used = 96.2Mb
6000000 | Size of Map = 2403537 | Capacity = 3,906.3Mb | Used = 115.8Mb
7000000 | Size of Map = 2796681 | Capacity = 3,906.3Mb | Used = 122.4Mb
8000000 | Size of Map = 3188915 | Capacity = 3,906.3Mb | Used = 127.3Mb
9000000 | Size of Map = 3586640 | Capacity = 3,906.3Mb | Used = 130.8Mb
10000000 | Size of Map = 3985190 | Capacity = 3,906.3Mb | Used = 134.4Mb
11000000 | Size of Map = 4379327 | Capacity = 3,906.3Mb | Used = 136.4Mb
12000000 | Size of Map = 4771262 | Capacity = 3,906.3Mb | Used = 139.5Mb
13000000 | Size of Map = 5157530 | Capacity = 3,906.3Mb | Used = 153.0Mb
14000000 | Size of Map = 5551225 | Capacity = 3,906.3Mb | Used = 153.4Mb
15000000 | Size of Map = 5940654 | Capacity = 3,906.3Mb | Used = 153.2Mb
16000000 | Size of Map = 6329560 | Capacity = 3,906.3Mb | Used = 152.7Mb
17000000 | Size of Map = 6720861 | Capacity = 3,906.3Mb | Used = 152.0Mb
18000000 | Size of Map = 7112984 | Capacity = 3,906.3Mb | Used = 151.1Mb
19000000 | Size of Map = 7498416 | Capacity = 3,906.3Mb | Used = 149.8Mb
20000000 | Size of Map = 7877277 | Capacity = 3,906.3Mb | Used = 164.3Mb
21000000 | Size of Map = 8264569 | Capacity = 3,906.3Mb | Used = 147.5Mb
22000000 | Size of Map = 8638660 | Capacity = 3,906.3Mb | Used = 146.1Mb
23000000 | Size of Map = 9009024 | Capacity = 3,906.3Mb | Used = 144.8Mb
24000000 | Size of Map = 9383993 | Capacity = 3,906.3Mb | Used = 159.7Mb
Exception in thread "Timer-0" java.lang.IllegalArgumentException: The
buffer
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$Merg
ingNioMemoryBuffer@56d93c0f seems not to belong to this allocator
at
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$Merg
ingNioMemoryBuffer.free(MergingByteBufferAllocator.java:396)
at
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator.free
(MergingByteBufferAllocator.java:157)
at
org.apache.directmemory.memory.MemoryManagerServiceImpl.free(MemoryManag
erServiceImpl.java:185)
at
org.apache.directmemory.memory.AbstractMemoryManager.free(AbstractMemory
Manager.java:149)
at
org.apache.directmemory.memory.AbstractMemoryManager.collectLFU(Abstract
MemoryManager.java:139)
at
org.apache.directmemory.cache.CacheServiceImpl.collectLFU(CacheServiceIm
pl.java:260)
at
org.apache.directmemory.cache.CacheServiceImpl$1.run(CacheServiceImpl.ja
va:85)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Exception in thread "main" java.lang.IllegalArgumentException: The
buffer
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$Merg
ingNioMemoryBuffer@37a2f588 seems not to belong to this allocator
at
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$Merg
ingNioMemoryBuffer.free(MergingByteBufferAllocator.java:396)
at
org.apache.directmemory.memory.allocator.MergingByteBufferAllocator.free
(MergingByteBufferAllocator.java:157)
at
org.apache.directmemory.memory.MemoryManagerServiceImpl.free(MemoryManag
erServiceImpl.java:185)
at
org.apache.directmemory.cache.CacheServiceImpl.store(CacheServiceImpl.ja
va:148)
at
org.apache.directmemory.cache.CacheServiceImpl.put(CacheServiceImpl.java
:118)
at
org.apache.directmemory.cache.CacheServiceImpl.put(CacheServiceImpl.java
:109)
at
com.datamentors.CreateVoltDBLoadFileAndMaps.<init>(CreateVoltDBLoadFileA
ndMaps.java:212)
at
com.datamentors.CreateVoltDBLoadFileAndMaps.main(CreateVoltDBLoadFileAnd
Maps.java:421)
Java Result: 1
BUILD SUCCESSFUL (total time: 8 minutes 3 seconds)
From: Scott Selvia [mailto:sselvia@datamentors.com]
Sent: Wednesday, April 16, 2014 3:13 PM
To: user@directmemory.apache.org
Subject: Loading Map hangs/boggs down once Initial Size is Reached
I'm loading a map with String, Integer values. The cacheservice is
initialized with the following code, the code works great until the
amount of memory "Used" reaches the initial size value when the cache is
declared. When the application starts the memory starts at 4 GB and
works it way up, at the point of bogging down the memory used by the
java process is 12 GB. The application is still running but it is very
slowwwwww.
AllocationPolicy allocationPolicy = new RoundRobinAllocationPolicy();
MemoryManagerService<Integer> memoryManager = new
MemoryManagerServiceImpl<>( allocationPolicy, true );
CacheService<String, Integer> cache = new DirectMemory<String,
Integer>().setMap(new ConcurrentSkipListMap<String,
Pointer<Integer>>()).setMemoryManager(memoryManager).setNumberOfBuffers(
4).setSize(Ram.Mb(1024)).setInitialCapacity(Ram.Mb(1024)).setConcurrency
Level(4).setDisposalTime((60 * 60 * 1000) *
3).setSerializer(SerializerFactory.createNewSerializer()).newCacheServic
e();
I've changed the code and simplified the cache creation line and took
out the disposal time, concurrency, and serializer. What I noticed when
I executed the code the amount of USED memory dropped and finally I got
a stacktrace. If items are being removed or rotated from the map, that
would not be good. I'm trying to build a unique list of items and I
don't want key value pairs rolling off the list.
CacheService<String, Integer> cache = new DirectMemory<String,
Integer>().setMap(new ConcurrentSkipListMap<String,
Pointer<Integer>>()).setMemoryManager(memoryManager).setNumberOfBuffers(
4).setSize(1024000000).newCacheService();
What is the best approach using DirectMemory to keep all of my items on
the list? There is not a lot of great information yet regarding best
practices, any help would be appreciated.
Scott Selvia
(O) 813.960.7800 x102
(F) 813.960.7811
http://www.datamentors.com <http://www.datamentors.com/>