[jira] [Commented] (HBASE-12116) Hot contention spots; writing

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493630#comment-14493630 ] 

stack commented on HBASE-12116:
-------------------------------

Esteban and I did some messing with a YCSB pure write load this afternoon. We got to a place where the four disks were saturated writing (CPU was not). Here's a few notes from the session:

+ Esteban found that upping ringbuffer slot count helped; default is 16k.. he was running at 512k (hbase.regionserver.wal.disruptor.event.count). Also said deferred sync helped too.
+ Would be good to figure how to do bigger writes so we get more on disk per op.
+ Adding to CSLM doPut, CRC, and compares are the big consumers of CPU (apart from G1GC) according to perf (see below), no surprise. Flight Recorder has CRC and unsafe compareto at 20% each, then CSLM#doPut at 12%.
+ Minors but interesting (according to FR) are ByteBloomFilter#set... could do with a tune up (the BB#gets are showing as 3.5% -- unsafe it). The murmur hashing shows up as a percent too (look at faster implementaitons -- guava?). We has ACL audit on. Thats 2% of CPU according to perf.

Perf top during run.
{code}
6.53%  perf-18890.map      [.] Ljava/util/concurrent/ConcurrentSkipListMap;.doPut
  5.78%  libjvm.so          [.] OtherRegionsTable::add_reference(void*, int)
  4.14%  perf-18890.map      [.] Lorg/apache/hadoop/util/PureJavaCrc32;.update
  3.27%  libjvm.so          [.] G1RemSet::refine_card(signed char*, int, bool)
  2.99%  libjvm.so          [.] G1ParCopyClosure<false, (G1Barrier)2, false>::copy_to_survivor_space(oopDesc*)
  2.90%  libjvm.so          [.] G1BlockOffsetArrayContigSpace::block_start_unsafe(void const*)
  2.69%  libjvm.so          [.] HeapRegion::oops_on_card_seq_iterate_careful(MemRegion, FilterOutOfRegionClosure*, bool, signed char*)
  2.66%  libc-2.12.so        [.] memcpy
  2.54%  perf-18890.map      [.] Lorg/apache/hadoop/hbase/KeyValue$KVComparator;.compare
  2.54%  libjvm.so          [.] void G1ParCopyClosure<false, (G1Barrier)2, false>::do_oop_work<unsigned int>(unsigned int*)
  2.09%  perf-18890.map      [.] jbyte_disjoint_arraycopy
  1.84%  libjvm.so          [.] instanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)
  1.71%  libjvm.so          [.] G1ParScanThreadState::trim_queue()
  1.63%  libjvm.so          [.] SparsePRT::add_card(int, int)
  1.43%  libjvm.so          [.] G1BlockOffsetArray::forward_to_block_containing_addr_const(HeapWord*, HeapWord*, void const*) const
  1.39%  perf-18890.map      [.] Lorg/apache/hadoop/hbase/util/Bytes$LexicographicalComparerHolder$UnsafeComparer;.compareTo
  1.35%  libc-2.12.so        [.] vfprintf
  1.23%  perf-18890.map      [.] Lorg/apache/hadoop/fs/FSDataOutputStream$PositionCache;.write
  1.12%  libjvm.so          [.] G1UpdateRSOrPushRefOopClosure::do_oop(unsigned int*)
  1.09%  libjvm.so          [.] instanceKlass::oop_oop_iterate_nv(oopDesc*, FilterOutOfRegionClosure*)
  1.09%  libjvm.so          [.] G1SATBCardTableModRefBS::mark_card_deferred(unsigned long)
  1.04%  libjvm.so          [.] HeapRegionDCTOC::walk_mem_region_with_cl(MemRegion, HeapWord*, HeapWord*, OopClosure*)
{code}

> Hot contention spots; writing
> -----------------------------
>
>                 Key: HBASE-12116
>                 URL: https://issues.apache.org/jira/browse/HBASE-12116
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>         Attachments: 12116.checkForReplicas.txt, 12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot 2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png, Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg
>
>
> Playing with flight recorder, here are some write-time contentious synchronizations/locks (picture coming)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)