You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2026/09/26 05:19:40 UTC
[kudu-CR] [util] an update on --never fsync flag usage
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24963
Change subject: [util] an update on --never_fsync flag usage
......................................................................
[util] an update on --never_fsync flag usage
This changelist extends --never_fsync flag's application into the
Posix{RW,Writrable}File::Flush() methods: they invoke sync_file_range()
system call, conceptually doing exactly the same what fsync() does for
a portion of a file [1]. The motivation for this came from experiments
of running a test workload generated by the `kudu perf loadgen` CLI tool
against a Kudu cluster that doesn't synchronize data to the underlying
storage at critical data points.
Also, wrap all FLAGS_never_fsync-based criteria into the EXPECT_FALSE()
macro since --never_fsync is a hidden/test-only flag which shouldn't be
ever enabled in a real production system.
[1] https://man7.org/linux/man-pages/man2/sync_file_range.2.html
Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
---
M src/kudu/util/env_posix.cc
1 file changed, 13 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/24963/1
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 3: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3072/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 19:18:47 +0000
Gerrit-HasComments: No
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.
Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Mon, 05 Oct 2026 05:39:13 +0000
Gerrit-HasComments: No
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 2:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3071/
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 01:11:44 +0000
Gerrit-HasComments: No
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 2: Verified+1
Build Successful
http://jenkins.kudu.apache.org/job/pre_commit/3071/ : SUCCESS
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 02:01:32 +0000
Gerrit-HasComments: No
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.
Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 2: Code-Review+1
(3 comments)
Looks good to me.
Just a couple nits. Feel free to ignore if you don't see any value in addressing those.
http://gerrit.cloudera.org:8080/#/c/24963/2/src/kudu/util/env-test.cc
File src/kudu/util/env-test.cc:
http://gerrit.cloudera.org:8080/#/c/24963/2/src/kudu/util/env-test.cc@1340
PS2, Line 1340: NSTANTIATE_TEST_SUITE_P(SyncFlags,
: FsSyncControlTest,
: ::testing::Combine(testing::Bool(), testing::Bool()));
nit: to improve readability of the test names, consider adding a name generator lambda instead of having opaque names. Something like this maybe:
++
strings::Substitute("NeverFsync$0_UseFsync$1",
std::get<0>(info.param) ? "True" : "False",
std::get<1>(info.param) ? "True" : "False");
++
http://gerrit.cloudera.org:8080/#/c/24963/2/src/kudu/util/env-test.cc@1358
PS2, Line 1358: FLUSH_SYNC
nit: here and at line 1376 - consider adding FLUSH_ASYNC variant of tests as well.
http://gerrit.cloudera.org:8080/#/c/24963/2/src/kudu/util/env-test.cc@1379
PS2, Line 1379:
nit: Maybe add a unit test for SyncDir() as well.
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 12:07:21 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3050/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 26 Sep 2026 06:02:50 +0000
Gerrit-HasComments: No
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 3:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3072/
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 18:27:23 +0000
Gerrit-HasComments: No
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Ashwani Raina, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24963
to look at the new patch set (#2).
Change subject: [util] an update on --never_fsync flag usage
......................................................................
[util] an update on --never_fsync flag usage
This changelist extends --never_fsync flag's application into the
Posix{RW,Writable}File::Flush() methods: they invoke sync_file_range()
system call, conceptually doing exactly the same what fsync() does, but
for a portion of a file [1]. The motivation for this came from a few
experiments running a test workload generated by the `kudu perf loadgen`
CLI tool against a Kudu cluster that doesn't synchronize data to the
underlying storage at critical data points.
In addition, introduce trace metric counters for call sites of the
sync_file_range() syscall and count in yet another fsync() invocation
on a directory in PosixEnv::SyncDir().
Also, wrap all FLAGS_never_fsync-based criteria into the PREDICT_FALSE()
macro since --never_fsync is a hidden/test-only flag which shouldn't be
ever enabled in a real production system.
[1] https://man7.org/linux/man-pages/man2/sync_file_range.2.html
Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
---
M src/kudu/util/env-test.cc
M src/kudu/util/env_posix.cc
2 files changed, 172 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/24963/2
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/24963/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24963/1//COMMIT_MSG@10
PS1, Line 10: Writrable
> typo: Writable
Done
http://gerrit.cloudera.org:8080/#/c/24963/1//COMMIT_MSG@17
PS1, Line 17: EXPECT_FALSE
> PREDICT_FALSE ?
Done
http://gerrit.cloudera.org:8080/#/c/24963/1/src/kudu/util/env_posix.cc
File src/kudu/util/env_posix.cc:
http://gerrit.cloudera.org:8080/#/c/24963/1/src/kudu/util/env_posix.cc@182
PS1, Line 182: DEFINE_bool(never_fsync, false,
: "Never fsync() anything to disk. This is used by certain test cases to "
: "speed up runtime. This is very unsafe to use in production.");
> The flag name 'never_fsync' can be quite confusing because it specifically
Right -- all those syscalls are about synchronizing the contents of the file with its run-time image, and they are simply a set of flavors of doing that for a particular use-case. There isn't much difference categorically.
It could be a good idea to update the name of the flag, but I'd rather do so in a separate changelist, indeed. Renaming the flag shouldn't be a big deal from the backwards compatibility perspective since it's a hidden/unsafe flag which is nobody should be using in real production systems. Also, I'd prefer something a bit narrower, e.g. never_fs_sync or no_explicit_fs_sync.
http://gerrit.cloudera.org:8080/#/c/24963/1/src/kudu/util/env_posix.cc@437
PS1, Line 437: DoSync
> There seems to be limited existing test coverage for this path as well as F
Right -- there isn't much coverage for this stuff since the original code came from LevelDB -- it's similar to other utilities imported into the Kudu's codebase.
Yes, I can add test coverage based on trace counters -- that's quite fragile and don't test actual functionality (i.e. checking whether there was a call to fsync(), or whether it was fdatasync(), but it could help in checking out the control flow paths.
Adding such tests satisfies formal criteria of 'having test coverage', but in general I'd think of better ways of testing this from the perspective of actual functionality. However, ad-hoc approach with trace counters/points should be good enough as a first step, indeed.
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 18:46:28 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 3: Verified+1
unrelated Java test failures in TSAN build config: known flakies such as org.apache.kudu.spark.kudu.DefaultSourceTes
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 19:49:11 +0000
Gerrit-HasComments: No
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.
Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/24963/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24963/1//COMMIT_MSG@17
PS1, Line 17: EXPECT_FALSE
PREDICT_FALSE ?
http://gerrit.cloudera.org:8080/#/c/24963/1/src/kudu/util/env_posix.cc
File src/kudu/util/env_posix.cc:
http://gerrit.cloudera.org:8080/#/c/24963/1/src/kudu/util/env_posix.cc@182
PS1, Line 182: DEFINE_bool(never_fsync, false,
: "Never fsync() anything to disk. This is used by certain test cases to "
: "speed up runtime. This is very unsafe to use in production.");
The flag name 'never_fsync' can be quite confusing because it specifically mentions fsync and leaves out the detail that it includes fsync(), fdatasync(), sync_file_range().
In other words, if 'never_fsync' is set to true, calls like DoSync(), Flush() and SyncDir() would exit early without calling fsync() or fdatasync() or sync_file_range(). So, essentially this should be something like 'never_sync'.
Changing the flag name along with the description would be the right course here but that expands the scope of this change, so an elaborate description for now should be worth doing.
What do you think?
http://gerrit.cloudera.org:8080/#/c/24963/1/src/kudu/util/env_posix.cc@437
PS1, Line 437: DoSync
There seems to be limited existing test coverage for this path as well as Flush(), SyncDir() w.r.t. FLAGS_never_fsync and FLAGS_env_use_fsync values.
Maybe add a couple unit tests in env-test.cc and make use of existing trace counters (fsync, fdatasync, etc) and add new one for 'sync_file_range' to test that appropriate paths are taken in accordance with the flag values.
Thoughts?
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 28 Sep 2026 07:03:31 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
[util] an update on --never_fsync flag usage
This changelist extends --never_fsync flag's application into the
Posix{RW,Writable}File::Flush() methods: they invoke sync_file_range()
system call, conceptually doing exactly the same what fsync() does, but
for a portion of a file [1]. The motivation for this came from a few
experiments running a test workload generated by the `kudu perf loadgen`
CLI tool against a Kudu cluster that doesn't synchronize data to the
underlying storage at critical data points.
In addition, introduce trace metric counters for call sites of the
sync_file_range() syscall and count in yet another fsync() invocation
on a directory in PosixEnv::SyncDir().
Also, wrap all FLAGS_never_fsync-based criteria into the PREDICT_FALSE()
macro since --never_fsync is a hidden/test-only flag which shouldn't be
ever enabled in a real production system.
[1] https://man7.org/linux/man-pages/man2/sync_file_range.2.html
Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Reviewed-on: http://gerrit.cloudera.org:8080/24963
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Ashwani Raina <ar...@gmail.com>
---
M src/kudu/util/env-test.cc
M src/kudu/util/env_posix.cc
2 files changed, 196 insertions(+), 18 deletions(-)
Approvals:
Alexey Serbin: Verified
Ashwani Raina: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3050/
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 26 Sep 2026 05:19:48 +0000
Gerrit-HasComments: No
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Ashwani Raina, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24963
to look at the new patch set (#3).
Change subject: [util] an update on --never_fsync flag usage
......................................................................
[util] an update on --never_fsync flag usage
This changelist extends --never_fsync flag's application into the
Posix{RW,Writable}File::Flush() methods: they invoke sync_file_range()
system call, conceptually doing exactly the same what fsync() does, but
for a portion of a file [1]. The motivation for this came from a few
experiments running a test workload generated by the `kudu perf loadgen`
CLI tool against a Kudu cluster that doesn't synchronize data to the
underlying storage at critical data points.
In addition, introduce trace metric counters for call sites of the
sync_file_range() syscall and count in yet another fsync() invocation
on a directory in PosixEnv::SyncDir().
Also, wrap all FLAGS_never_fsync-based criteria into the PREDICT_FALSE()
macro since --never_fsync is a hidden/test-only flag which shouldn't be
ever enabled in a real production system.
[1] https://man7.org/linux/man-pages/man2/sync_file_range.2.html
Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
---
M src/kudu/util/env-test.cc
M src/kudu/util/env_posix.cc
2 files changed, 196 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/24963/3
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/24963/2/src/kudu/util/env-test.cc
File src/kudu/util/env-test.cc:
http://gerrit.cloudera.org:8080/#/c/24963/2/src/kudu/util/env-test.cc@1340
PS2, Line 1340: NSTANTIATE_TEST_SUITE_P(SyncFlags,
: FsSyncControlTest,
: ::testing::Combine(testing::Bool(), testing::Bool()));
> nit: to improve readability of the test names, consider adding a name gener
Done
http://gerrit.cloudera.org:8080/#/c/24963/2/src/kudu/util/env-test.cc@1358
PS2, Line 1358: FLUSH_SYNC
> nit: here and at line 1376 - consider adding FLUSH_ASYNC variant of tests a
IMO this is out of the scope of this changelist. It may be a good venue for a separate changelist which adds test coverage for Env::{RW,Writable}File flush modes' behavior.
http://gerrit.cloudera.org:8080/#/c/24963/2/src/kudu/util/env-test.cc@1379
PS2, Line 1379:
> nit: Maybe add a unit test for SyncDir() as well.
Done
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 18:27:29 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 1:
(1 comment)
Once Ashwani`s comments are cleared up this looks good to me.
http://gerrit.cloudera.org:8080/#/c/24963/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24963/1//COMMIT_MSG@10
PS1, Line 10: Writrable
typo: Writable
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 10:42:14 +0000
Gerrit-HasComments: Yes
[kudu-CR] [util] an update on --never fsync flag usage
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/24963 )
Change subject: [util] an update on --never_fsync flag usage
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24963
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I637e3bd98da98495a38f78176a4a862ab6ae8dd3
Gerrit-Change-Number: 24963
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 00:34:42 +0000
Gerrit-HasComments: No