You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Adar Dembo (JIRA)" <ji...@apache.org> on 2017/06/22 22:27:00 UTC
[jira] [Commented] (KUDU-2052) Use XFS_IOC_UNRESVSP64 ioctl to
punch holes on xfs filesystems
[ https://issues.apache.org/jira/browse/KUDU-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060114#comment-16060114 ]
Adar Dembo commented on KUDU-2052:
----------------------------------
The following experiments were run using a single spinning HDD on an CentOS 6.6 machine. First, a filesystem was created in a 10G file and mounted with -o loop. Inside the filesystem a 1G test file was created using dd from /dev/urandom. The first 100M of the test file were punched out via fallocate, sync was run, and then the following repunching tests in a loop using perf stat.
ext4 with fallocate-based hole punching:
{noformat}
Performance counter stats for 'fallocate -p -o 0 -l 100M foo' (1000 runs):
0.269927 task-clock # 0.635 CPUs utilized ( +- 0.33% )
0 context-switches # 0.000 K/sec
0 cpu-migrations # 0.041 K/sec ( +- 30.00% )
150 page-faults # 0.555 M/sec ( +- 0.01% )
770,390 cycles # 2.854 GHz ( +- 0.33% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
477,294 instructions # 0.62 insns per cycle ( +- 0.04% )
98,535 branches # 365.044 M/sec ( +- 0.03% )
3,979 branch-misses # 4.04% of all branches ( +- 0.61% )
0.000425207 seconds time elapsed ( +- 0.45% )
{noformat}
xfs filesystem with fallocate-based hole punching:
{noformat}
Performance counter stats for 'fallocate -p -o 0 -l 100M foo' (1000 runs):
0.403296 task-clock # 0.013 CPUs utilized ( +- 0.32% )
2 context-switches # 0.005 M/sec ( +- 0.17% )
0 cpu-migrations # 0.017 K/sec ( +- 37.68% )
150 page-faults # 0.371 M/sec ( +- 0.01% )
1,112,706 cycles # 2.759 GHz ( +- 0.17% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
505,027 instructions # 0.45 insns per cycle ( +- 0.03% )
103,652 branches # 257.013 M/sec ( +- 0.02% )
5,750 branch-misses # 5.55% of all branches ( +- 0.04% )
0.031220273 seconds time elapsed ( +- 0.57% )
{noformat}
xfs filesystem with XFS_IOC_UNRESVSP64-based hole punching:
{noformat}
Performance counter stats for 'xfs_io -c unresvsp 0 104857600 foo' (1000 runs):
0.477930 task-clock # 0.677 CPUs utilized ( +- 0.28% )
0 context-switches # 0.004 K/sec ( +- 70.68% )
0 cpu-migrations # 0.017 K/sec ( +- 39.42% )
215 page-faults # 0.449 M/sec ( +- 0.01% )
1,463,629 cycles # 3.062 GHz ( +- 0.15% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
1,150,346 instructions # 0.79 insns per cycle ( +- 0.01% )
241,338 branches # 504.964 M/sec ( +- 0.01% )
9,753 branch-misses # 4.04% of all branches ( +- 0.02% )
0.000706070 seconds time elapsed ( +- 0.36% )
{noformat}
> Use XFS_IOC_UNRESVSP64 ioctl to punch holes on xfs filesystems
> --------------------------------------------------------------
>
> Key: KUDU-2052
> URL: https://issues.apache.org/jira/browse/KUDU-2052
> Project: Kudu
> Issue Type: Bug
> Components: util
> Affects Versions: 1.4.0
> Reporter: Adar Dembo
> Assignee: Adar Dembo
> Priority: Critical
>
> One of the changes in Kudu 1.4 is a more comprehensive repair functionality in log block manager startup. Amongst other things this includes a heuristic to detect whether an LBM container consumes more disk space than it should, based on the live blocks in the container. If the heuristic fires, the LBM reclaims the extra disk space by truncating the end of the container and repunching out all of the dead blocks in the container.
> We brought up Kudu 1.4 on a large production cluster running xfs and observed pathologically slow startup times. On one node, there was a three hour gap between the last bit of data directory processing and the end of LBM startup in general. This time can only be attributed to hole repunching, which is executed by the same set of thread pools that open the data directories.
> Further research revealed that on xfs in el6, a hole punch via fallocate() _always_ includes an fsync() (in the kernel), even if the underlying data was already punched out. This isn't the case with ext4, nor does it appear to be the case with xfs in more modern kernels (though this hasn't been confirmed).
> xfs provides the [XFS_IOC_UNRESVSP64 ioctl|https://linux.die.net/man/3/xfsctl], which can be used to deallocate space from a file. That sounds an awful lot like hole punching, and some quick performance tests show that it doesn't incur the cost of an fsync(). We should switch over to it when punching holes on xfs. Certainly on older (i.e. el6) kernels, and potentially everywhere for simplicity's sake.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)