slowdown in ingest performance

[Ara Ebrahimi <ar...@argyledata.com>]

Hi,

I’m trying to tune an accumulo 1.7 cluster. We have 3 tablet servers. tservers are configured to use 32GB of heap (each machine has 256GB mem and 4 disks). These are the configuration changes we’ve made:

config -s table.cache.block.enable=true
config -s tserver.cache.data.size=4G

config -s tserver.compaction.minor.concurrent.max=50
config -s tserver.compaction.major.concurrent.max=8
config -s tserver.recovery.concurrent.max=4

config -s tserver.wal.replication=2
config -s tserver.wal.blocksize=2G
config -s tserver.walog.max.size=2G
config -s tserver.mutation.queue.max=4M
config -s tserver.memory.maps.max=20G

config -s table.compaction.major.ratio=10
config -s table.compaction.minor.logs.threshold=15
config -s table.file.max=30

config -s table.split.threshold=1024G

config -s table.durability=flush

Reasonable? Any suggestions?

The problem is we see a gradual ingest slow down (see attached screenshot). Why the slowdown? And how to prevent it? The ingest process is one backed by Kafka and is super fast.

Thanks,
Ara.

[cid:2CCF63FA-4C75-42E6-872F-CF051EF42983]



________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.

________________________________

Re: slowdown in ingest performance

[Keith Turner <ke...@deenlo.com>]

On Tue, Jan 10, 2017 at 4:25 PM, Ara Ebrahimi <ar...@argyledata.com>
wrote:

> Hi,
>
> I’m trying to tune an accumulo 1.7 cluster. We have 3 tablet servers.
> tservers are configured to use 32GB of heap (each machine has 256GB mem and
> 4 disks). These are the configuration changes we’ve made:
>
> config -s table.cache.block.enable=true
> config -s tserver.cache.data.size=4G
>
> config -s tserver.compaction.minor.concurrent.max=50
> config -s tserver.compaction.major.concurrent.max=8
> config -s tserver.recovery.concurrent.max=4
>
> config -s tserver.wal.replication=2
> config -s tserver.wal.blocksize=2G
> config -s tserver.walog.max.size=2G
> config -s tserver.mutation.queue.max=4M
> config -s tserver.memory.maps.max=20G
>
> config -s table.compaction.major.ratio=10
> config -s table.compaction.minor.logs.threshold=15
> config -s table.file.max=30
>
> config -s table.split.threshold=1024G
>
> config -s table.durability=flush
>
> Reasonable? Any suggestions?
>

One additional thing to try would be to change the metadata table
durability settings as I suggested in the following blog post.

http://accumulo.apache.org/blog/2016/11/02/durability-performance.html#configuring-wal-flushsync-in-accumulo-17

Below are the settings from the blog post.

  config -s table.durability=flush
  config -t accumulo.metadata -d table.durability
  config -t accumulo.root -d table.durability

Also are you using 1.7.2?   The blog post mentions ACCUMULO-4112 as fixing
a cause of calls to hsync to 1.7.2.


>
> The problem is we see a gradual ingest slow down (see attached
> screenshot). Why the slowdown? And how to prevent it? The ingest process is
> one backed by Kafka and is super fast.
>
> Thanks,
> Ara.
>
>
>
>
> ------------------------------
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
> ------------------------------
>