RAM high consume

[Eduardo Costa Alfaia <e....@unibs.it>]

      Hi all,

I am doing some tests using JavaNetworkWordcount and I have some 
questions about the performance machine, my tests' time are 
approximately 2 min.

Why does the RAM Memory decrease meaningly? I have done tests with 2, 3 
machines and I had gotten the same behavior.

What should I do to get a better performance in this case?


##### Star Test

computer1
              total       used       free     shared    buffers cached
Mem:          3945        711 3233 0          3        430
-/+ buffers/cache:        276       3668
Swap:            0          0          0

  14:42:50 up 73 days,  3:32,  2 users,  load average: 0.00, 0.06, 0.21


14/04/04 14:24:56 INFO BlockManagerMasterActor$BlockManagerInfo: Added 
input-0-1396614314400 in memory on computer1.ant-net:60820 (size: 826.1 
KB, free: 542.9 MB)
14/04/04 14:24:56 INFO MemoryStore: ensureFreeSpace(845956) called with 
curMem=47278100, maxMem=825439027
14/04/04 14:24:56 INFO MemoryStore: Block input-0-1396614314400 stored 
as bytes to memory (size 826.1 KB, free 741.3 MB)
14/04/04 14:24:56 INFO BlockManagerMasterActor$BlockManagerInfo: Added 
input-0-1396614314400 in memory on computer8.ant-net:49743 (size: 826.1 
KB, free: 741.3 MB)
14/04/04 14:24:56 INFO BlockManagerMaster: Updated info of block 
input-0-1396614314400
14/04/04 14:24:56 INFO TaskSetManager: Finished TID 272 in 84 ms on 
computer1.ant-net (progress: 0/1)
14/04/04 14:24:56 INFO TaskSchedulerImpl: Remove TaskSet 43.0 from pool
14/04/04 14:24:56 INFO DAGScheduler: Completed ResultTask(43, 0)
14/04/04 14:24:56 INFO DAGScheduler: Stage 43 (take at 
DStream.scala:594) finished in 0.088 s
14/04/04 14:24:56 INFO SparkContext: Job finished: take at 
DStream.scala:594, took 1.872875734 s
-------------------------------------------
Time: 1396614289000 ms
-------------------------------------------
(Santiago,1)
(liveliness,1)
(Sun,1)
(reapers,1)
(offer,3)
(BARBER,3)
(shrewdness,1)
(truism,1)
(hits,1)
(merchant,1)



##### End Test
computer1
              total       used       free     shared    buffers cached
Mem:          3945       2209 1735 0          5        773
-/+ buffers/cache:       1430       2514
Swap:            0          0          0

  14:46:05 up 73 days,  3:35,  2 users,  load average: 2.69, 1.07, 0.55


14/04/04 14:26:57 INFO TaskSetManager: Starting task 183.0:0 as TID 696 
on executor 0: computer1.ant-net (PROCESS_LOCAL)
14/04/04 14:26:57 INFO TaskSetManager: Serialized task 183.0:0 as 1981 
bytes in 0 ms
14/04/04 14:26:57 INFO MapOutputTrackerMasterActor: Asked to send map 
output locations for shuffle 81 to spark@computer1.ant-net:44817
14/04/04 14:26:57 INFO MapOutputTrackerMaster: Size of output statuses 
for shuffle 81 is 212 bytes
14/04/04 14:26:57 INFO BlockManagerMasterActor$BlockManagerInfo: Added 
input-0-1396614336600 on disk on computer1.ant-net:60820 (size: 1441.7 KB)
14/04/04 14:26:57 INFO BlockManagerMasterActor$BlockManagerInfo: Added 
input-0-1396614435200 in memory on computer1.ant-net:60820 (size: 1295.7 
KB, free: 589.3 KB)
14/04/04 14:26:57 INFO TaskSetManager: Finished TID 696 in 56 ms on 
computer1.ant-net (progress: 0/1)
14/04/04 14:26:57 INFO TaskSchedulerImpl: Remove TaskSet 183.0 from pool
14/04/04 14:26:57 INFO DAGScheduler: Completed ResultTask(183, 0)
14/04/04 14:26:57 INFO DAGScheduler: Stage 183 (take at 
DStream.scala:594) finished in 0.057 s
14/04/04 14:26:57 INFO SparkContext: Job finished: take at 
DStream.scala:594, took 1.575268894 s
-------------------------------------------
Time: 1396614359000 ms
-------------------------------------------
(hapless,9)
(reapers,8)
(amazed,113)
(feebleness,7)
(offer,148)
(rabble,27)
(exchanging,7)
(merchant,20)
(incentives,2)
(quarrel,48)
...


Thanks Guys

-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155