[jira] [Created] (HBASE-29107) shell: Improve 'count' performance

["Junegunn Choi (Jira)" <ji...@apache.org>]

Junegunn Choi created HBASE-29107:
-------------------------------------

             Summary: shell: Improve 'count' performance
                 Key: HBASE-29107
                 URL: https://issues.apache.org/jira/browse/HBASE-29107
             Project: HBase
          Issue Type: Improvement
          Components: shell
            Reporter: Junegunn Choi


I propose two changes to the 'count' command of HBase shell to improve its performance.
h2. Not setting scanner caching

The command currently sets the scanner caching to 10 rows by default, and instructs the users to increase it if necessary. According to HBASE-2331, the default value was chosen as such in case the table has large records.
{quote}Default value of 10 is really slow, but should be kept as low for clients with huge rows.
{quote}
However, with the current version of HBase, we use a better mechanism {{{}hbase.client.scanner.max.result.size{}}}, which is 2MB by default. So just by not setting the scanner caching, we automatically get a better performance, and we don't have to worry about huge rows.
h3. Test
{code:java}
# Create table
create 't', 'd', {NUMREGIONS => 4, SPLITALGO => 'HexStringSplit'}

# Insert data
data = '_' * 1024
bm = @hbase.connection.getBufferedMutator(TableName.valueOf('t'))
(8 * 1024 * 1024).times do |i|
  row = format('%010x', i).reverse.to_java_bytes
  p = org.apache.hadoop.hbase.client.Put.new(row)
  p.addColumn('d'.to_java_bytes, ''.to_java_bytes, data.to_java_bytes)
  bm.mutate(p)
end
bm.close

# Before patch
count 't', INTERVAL => 100000
  # 8388608 row(s)
  # Took 53.5826 seconds

# Before patch with custom 'CACHE'
count 't', INTERVAL => 100000, CACHE => 2000
  # 8388608 row(s)
  # Took 13.6717 seconds

# After patch
count 't', INTERVAL => 100000
  # 8388608 row(s)
  # Took 14.0911 seconds
{code}
The test was performed locally on my machine, so the different in performance in a real cluster should be larger.
h2. KeyOnlyFilter

Another thing we can do is to apply {{KeyOnlyFilter}} as well because we're not interested in the values. This helps when the records are large.
h3. Test
{code:java}
create 't2', 'd', {NUMREGIONS => 4, SPLITALGO => 'HexStringSplit'}

data = '_' * 1024 * 1024
bm = @hbase.connection.getBufferedMutator(TableName.valueOf('t2'))
(8 * 1024).times do |i|
  row = format('%010x', i).reverse.to_java_bytes
  p = org.apache.hadoop.hbase.client.Put.new(row)
  p.addColumn('d'.to_java_bytes, ''.to_java_bytes, data.to_java_bytes)
  bm.mutate(p)
end
bm.close

# Before patch
count 't2'
  # 8192 row(s)
  # Took 8.8952 seconds

# After patch
count 't2'
  # 8192 row(s)
  # Took 3.4052 seconds
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)