[jira] [Commented] (KUDU-3318) Log Block Container metadata consumed too much disk space

["Yingchun Lai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17413461#comment-17413461 ] 

Yingchun Lai commented on KUDU-3318:
------------------------------------

A easy way to resolve this problem si to add a limitation to .metadata file too, when it reach that limit size(similar to data file reach its limit offset, or block number reach its number limit), the container is refused to append more blocks, and then after all blocks are deleted, the whole container will be removed.

> Log Block Container metadata consumed too much disk space
> ---------------------------------------------------------
>
>                 Key: KUDU-3318
>                 URL: https://issues.apache.org/jira/browse/KUDU-3318
>             Project: Kudu
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Yingchun Lai
>            Priority: Major
>
> In log block container, blocks in .data file are append only, there is a related append only .metadata file to trace blocks in .data, this type of entries in metadata are in CREATE type, the other type of entries in metadata are type of DELETE, it means mark the corresponding CREATE block as deleted.
> If there is a pair of CREATE and DELETE entries of a same block id, LBM use hole punch to reclaim disk space in .data file, but the entries in .metadata will not be compacted except bootstrap.
> Another way to limit metadata is the .data file offset reach its size limitation(default 10GB), or block number in metadata reach its limitation(no limit on default).
> I found a case in product environment that metadata consumed too many disk space and near to .data's disk space, it's a waste, and make users confused and complain that the actual disk space is far more than user's data.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)