You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2019/11/13 07:17:00 UTC
[jira] [Commented] (KUDU-2977) Sharding block map of
LogBlockManager
[ https://issues.apache.org/jira/browse/KUDU-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973074#comment-16973074 ]
ASF subversion and git services commented on KUDU-2977:
-------------------------------------------------------
Commit 98f44f4537ceddffedaf9afce26b634c4ab2142a in kudu's branch refs/heads/master from Yingchun Lai
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=98f44f4 ]
KUDU-2977 Sharding block map to speed up tserver startup
Separate LogBlockManager's block lock and container lock,
and sharding block map.
Tablet server with multiple data directories cost much time to
startup, there is a thread for each data directory, each of them
should acquire a global lock to add blocks found in its own data
directory to the global block map, so there is a queue to acquire
lock that may cost much time. The same when remove a batch of blocks.
After bunch of benchmarks, we found pretty stable improvment of this
patch, time saving much more when block count increasing (reduce about
20% startup time when 10,000,000 blocks on 8 data directories).
Sharding count is 8, which has the best performance after comparing
with 1, 2, 4, 8, 16, 32, 64 and 128.
Change-Id: If0d5c13e051a2c1d6cfd1c9ad7db8a3cd195459d
Reviewed-on: http://gerrit.cloudera.org:8080/14555
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
> Sharding block map of LogBlockManager
> -------------------------------------
>
> Key: KUDU-2977
> URL: https://issues.apache.org/jira/browse/KUDU-2977
> Project: Kudu
> Issue Type: Improvement
> Components: fs
> Reporter: Yingchun Lai
> Assignee: Yingchun Lai
> Priority: Major
>
> I found LBM may manage a very large block map when tserver hold large scale data, metric like:
> {code:java}
> {
> "name": "log_block_manager_blocks_under_management",
> "value": 7014790
> }
> {code}
> LogBlock get/add should hold map's lock first, this is costly and will slow down statup time.
> [~tlipcon] also mentioned in https://issues.apache.org/jira/browse/KUDU-2014.
>
> We can improve it by sharding this big map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)