You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-user@hadoop.apache.org by Сергей Казаков <ss...@gmail.com> on 2015/05/06 16:02:48 UTC

BlockPoolSliceScanner and GC

Hi everyone!


We've been investigating some unpredictably long GC pauses in Hadoop
Datanode (2.5.1, Oracle JRE8_25) and noticed strange behaviour in
DataBlockScanner.

DataBlockScanner initiates block verification tasks during DN startup and
every 504 hours after. It holds some structures in memory heap (HashMap,
TreeSet & LightWeightGSet) which, in our case, are around 330Mb total (over
1.5m blocks). Are these structures useful after block verification task
completes? Tenuring and promoting these objects to OldGen causes some
terrible STW pauses we would like to avoid.


Kind regards,
Sergey Kazakov