You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2022/04/18 01:45:00 UTC
[GitHub] [bookkeeper] hangc0276 opened a new pull request, #3212: Added flag to control whether to transition to read-only mode when any disks full
hangc0276 opened a new pull request, #3212:
URL: https://github.com/apache/bookkeeper/pull/3212
### Motivation
Fix #3141
If we configure multi ledger disks, the bookie will transition to read-only mode when all disks are reached max usage threshold. However, if just one ledger disk reached max usage threshold, the bookie will continue to receive write requests, which will lead to the ledger disk out of space.
### Changes
Add a flag to control whether to transition to read-only mode when any disks reaches max usage threshold, and when all disks recovered to less than max usage threshold, the bookie will transition to read-write mode.
I will add a test soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] dlg99 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
dlg99 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1110029390
Overall: LGTM.
I still insist on updating config file with description of the feature + the docs.
@hangc0276 I hear your point about the length of the file but there are other things to consider:
* On many occasions doc were out of sync and config works a s a more up-to-date version of the doc
* config is easy to diff against an existing prod config to check if there are any new features one wants to try, how the current one is different form defaults etc. Diffing code is a bit more painful, especially considering that here is ServerConfiguration, AbstractConfiguration etc
* new user is unlikely to start digging into the code to look for the config options
* currently community agreement, as I remember, is to add new options and their description into the config file. You can start a discussion on ML to challenge this but we'll need to come up with a better solution for options discoverability than forcing users to dig into the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on a diff in pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on code in PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#discussion_r857122652
##########
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/LedgerDirsManagerTest.java:
##########
@@ -240,6 +240,76 @@ private void testLedgerDirsMonitorDuringTransition(boolean highPriorityWritesAll
assertTrue(mockLedgerDirsListener.highPriorityWritesAllowed);
}
+ @Test
+ public void testIsReadOnlyModeOnAnyDiskFullEnabled() throws Exception {
+ testAnyLedgerFullTransitToReadOnly(true);
+ testAnyLedgerFullTransitToReadOnly(false);
+ }
+
+ public void testAnyLedgerFullTransitToReadOnly(boolean isReadOnlyModeOnAnyDiskFullEnabled) throws Exception {
+ ledgerMonitor.shutdown();
+
+ final float nospace = 0.90f;
+ final float lwm = 0.80f;
+ HashMap<File, Float> usageMap;
+
+ File tmpDir1 = createTempDir("bkTest", ".dir");
+ File curDir1 = BookieImpl.getCurrentDirectory(tmpDir1);
+ BookieImpl.checkDirectoryStructure(curDir1);
+
+ File tmpDir2 = createTempDir("bkTest", ".dir");
+ File curDir2 = BookieImpl.getCurrentDirectory(tmpDir2);
+ BookieImpl.checkDirectoryStructure(curDir2);
+
+ conf.setDiskUsageThreshold(nospace);
+ conf.setDiskLowWaterMarkUsageThreshold(lwm);
+ conf.setDiskUsageWarnThreshold(nospace);
+ conf.setReadOnlyModeOnAnyDiskFullEnabled(isReadOnlyModeOnAnyDiskFullEnabled);
+ conf.setLedgerDirNames(new String[] { tmpDir1.toString(), tmpDir2.toString() });
+
+ mockDiskChecker = new MockDiskChecker(nospace, warnThreshold);
+ dirsManager = new LedgerDirsManager(conf, conf.getLedgerDirs(),
+ new DiskChecker(conf.getDiskUsageThreshold(), conf.getDiskUsageWarnThreshold()), statsLogger);
+ ledgerMonitor = new LedgerDirsMonitor(conf, mockDiskChecker, Collections.singletonList(dirsManager));
+ usageMap = new HashMap<>();
+ usageMap.put(curDir1, 0.1f);
+ usageMap.put(curDir2, 0.1f);
+ mockDiskChecker.setUsageMap(usageMap);
+ ledgerMonitor.init();
+ final MockLedgerDirsListener mockLedgerDirsListener = new MockLedgerDirsListener();
+ dirsManager.addLedgerDirsListener(mockLedgerDirsListener);
+ ledgerMonitor.start();
+
+ Thread.sleep((diskCheckInterval * 2) + 100);
+ assertFalse(mockLedgerDirsListener.readOnly);
Review Comment:
In BookKeeper code, there are a lot of `Thread.sleep`. I will use another PR to introduce awaitility to make all tests get ride of `Thread.sleep`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1112154538
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1111675467
> Overall: LGTM.
>
> I still insist on updating config file with description of the feature + the docs.
>
> @hangc0276 I hear your point about the length of the file but there are other things to consider:
>
> * On many occasions doc were out of sync and config works a s a more up-to-date version of the doc
> * config is easy to diff against an existing prod config to check if there are any new features one wants to try, how the current one is different form defaults etc. Diffing code is a bit more painful, especially considering that here is ServerConfiguration, AbstractConfiguration etc
> * new user is unlikely to start digging into the code to look for the config options
> * currently community agreement, as I remember, is to add new options and their description into the config file. You can start a discussion on ML to challenge this but we'll need to come up with a better solution for options discoverability than forcing users to dig into the code.
@dlg99 I have update the doc, Please help take a look, thanks a lot. I wonder if we can include this Pr into 4.15.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1109412096
> I'll look at the change a little bit later, but the first thing I noticed is that the new config option is not added to/not documented at the `conf/bk_server.conf` and not added to the docs.
IMO, we'd better to keep `conf/bk_server.conf` simple and just include import config. For advanced users who want to change chose config item, they can get the description from `ServerConfiguration.java`. Just like Apache Pulsar, we put all the config item into the `conf/broker.conf`, leading to nearly 1k lines of the configuration file. For common users, they will be confused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1111750516
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] leizhiyuan commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
leizhiyuan commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1193042969
thanks for your work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on a diff in pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#discussion_r857118720
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDirsMonitor.java:
##########
@@ -68,6 +68,10 @@ public LedgerDirsMonitor(final ServerConfiguration conf,
private void check(final LedgerDirsManager ldm) {
final ConcurrentMap<File, Float> diskUsages = ldm.getDiskUsages();
+ boolean anyDiskFulled = false;
+ boolean highPriorityWritesAllowed = true;
+ boolean anyDiskRecovered = false;
Review Comment:
someDiskRecovered sounds better
##########
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/LedgerDirsManagerTest.java:
##########
@@ -240,6 +240,76 @@ private void testLedgerDirsMonitorDuringTransition(boolean highPriorityWritesAll
assertTrue(mockLedgerDirsListener.highPriorityWritesAllowed);
}
+ @Test
+ public void testIsReadOnlyModeOnAnyDiskFullEnabled() throws Exception {
+ testAnyLedgerFullTransitToReadOnly(true);
+ testAnyLedgerFullTransitToReadOnly(false);
+ }
+
+ public void testAnyLedgerFullTransitToReadOnly(boolean isReadOnlyModeOnAnyDiskFullEnabled) throws Exception {
+ ledgerMonitor.shutdown();
+
+ final float nospace = 0.90f;
+ final float lwm = 0.80f;
+ HashMap<File, Float> usageMap;
+
+ File tmpDir1 = createTempDir("bkTest", ".dir");
+ File curDir1 = BookieImpl.getCurrentDirectory(tmpDir1);
+ BookieImpl.checkDirectoryStructure(curDir1);
+
+ File tmpDir2 = createTempDir("bkTest", ".dir");
+ File curDir2 = BookieImpl.getCurrentDirectory(tmpDir2);
+ BookieImpl.checkDirectoryStructure(curDir2);
+
+ conf.setDiskUsageThreshold(nospace);
+ conf.setDiskLowWaterMarkUsageThreshold(lwm);
+ conf.setDiskUsageWarnThreshold(nospace);
+ conf.setReadOnlyModeOnAnyDiskFullEnabled(isReadOnlyModeOnAnyDiskFullEnabled);
+ conf.setLedgerDirNames(new String[] { tmpDir1.toString(), tmpDir2.toString() });
+
+ mockDiskChecker = new MockDiskChecker(nospace, warnThreshold);
+ dirsManager = new LedgerDirsManager(conf, conf.getLedgerDirs(),
+ new DiskChecker(conf.getDiskUsageThreshold(), conf.getDiskUsageWarnThreshold()), statsLogger);
+ ledgerMonitor = new LedgerDirsMonitor(conf, mockDiskChecker, Collections.singletonList(dirsManager));
+ usageMap = new HashMap<>();
+ usageMap.put(curDir1, 0.1f);
+ usageMap.put(curDir2, 0.1f);
+ mockDiskChecker.setUsageMap(usageMap);
+ ledgerMonitor.init();
+ final MockLedgerDirsListener mockLedgerDirsListener = new MockLedgerDirsListener();
+ dirsManager.addLedgerDirsListener(mockLedgerDirsListener);
+ ledgerMonitor.start();
+
+ Thread.sleep((diskCheckInterval * 2) + 100);
+ assertFalse(mockLedgerDirsListener.readOnly);
Review Comment:
Can we use awaitility here?
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDirsMonitor.java:
##########
@@ -68,6 +68,10 @@ public LedgerDirsMonitor(final ServerConfiguration conf,
private void check(final LedgerDirsManager ldm) {
final ConcurrentMap<File, Float> diskUsages = ldm.getDiskUsages();
+ boolean anyDiskFulled = false;
Review Comment:
What about 'SomeDiskFull'
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java:
##########
@@ -2395,6 +2396,27 @@ public boolean isReadOnlyModeEnabled() {
return getBoolean(READ_ONLY_MODE_ENABLED, true);
}
+ /**
+ * Set whether the bookie is able to go into read-only mode when any disk is full.
+ * If this set to false, it will behave to READ_ONLY_MODE_ENABLED flag.
+ *
+ * @param enabled whether to enable read-only mode when any disk is full.
+ * @return
+ */
+ public ServerConfiguration setReadOnlyModeOnAnyDiskFullEnabled(boolean enabled) {
+ setProperty(READ_ONLY_MODE_ON_ANY_DISK_FULL_ENABLED, enabled);
+ return this;
+ }
+
+ /**
+ * Get whether read-only mode is enable when any disk is full. The default is false.
+ *
+ * @return boolean
+ */
+ public boolean isReadOnlyModeOnAnyDiskFullEnabled() {
+ return getBoolean(READ_ONLY_MODE_ON_ANY_DISK_FULL_ENABLED, false);
Review Comment:
In this case it is better to set this to true.
It is a safe feature and if we don't enable it by default nobody will leverage it
I wonder if we need a flag at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on a diff in pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on code in PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#discussion_r857122376
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java:
##########
@@ -2395,6 +2396,27 @@ public boolean isReadOnlyModeEnabled() {
return getBoolean(READ_ONLY_MODE_ENABLED, true);
}
+ /**
+ * Set whether the bookie is able to go into read-only mode when any disk is full.
+ * If this set to false, it will behave to READ_ONLY_MODE_ENABLED flag.
+ *
+ * @param enabled whether to enable read-only mode when any disk is full.
+ * @return
+ */
+ public ServerConfiguration setReadOnlyModeOnAnyDiskFullEnabled(boolean enabled) {
+ setProperty(READ_ONLY_MODE_ON_ANY_DISK_FULL_ENABLED, enabled);
+ return this;
+ }
+
+ /**
+ * Get whether read-only mode is enable when any disk is full. The default is false.
+ *
+ * @return boolean
+ */
+ public boolean isReadOnlyModeOnAnyDiskFullEnabled() {
+ return getBoolean(READ_ONLY_MODE_ON_ANY_DISK_FULL_ENABLED, false);
Review Comment:
If we enable is feature, it changed the current behavior. I wonder if we make the default false first, and send an email to dev mail list to discuss it. If approved, I use another PR to change the default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1107680343
ping @eolivelli @dlg99 @Vanlightly @nicoloboschi Please help take a look, thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] dlg99 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
dlg99 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1109407688
I'll look at the change a little bit later, but the first thing I noticed is that the new config option is not added to/not documented at the `conf/bk_server.conf` and not added to the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1110536417
> Overall: LGTM.
>
> I still insist on updating config file with description of the feature + the docs.
>
> @hangc0276 I hear your point about the length of the file but there are other things to consider:
>
> * On many occasions doc were out of sync and config works a s a more up-to-date version of the doc
> * config is easy to diff against an existing prod config to check if there are any new features one wants to try, how the current one is different form defaults etc. Diffing code is a bit more painful, especially considering that here is ServerConfiguration, AbstractConfiguration etc
> * new user is unlikely to start digging into the code to look for the config options
> * currently community agreement, as I remember, is to add new options and their description into the config file. You can start a discussion on ML to challenge this but we'll need to come up with a better solution for options discoverability than forcing users to dig into the code.
@dlg99 Thanks for your clarification. I will add config item in `conf/bookkeeper.conf` and add doc into website. And then I will start a discussion on mail list about the configuration issue that I concerned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] merlimat merged pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
merlimat merged PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on a diff in pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on code in PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#discussion_r857122019
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDirsMonitor.java:
##########
@@ -68,6 +68,10 @@ public LedgerDirsMonitor(final ServerConfiguration conf,
private void check(final LedgerDirsManager ldm) {
final ConcurrentMap<File, Float> diskUsages = ldm.getDiskUsages();
+ boolean anyDiskFulled = false;
Review Comment:
make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1107868827
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3212: Added flag to control whether to transition to read-only mode when any disks full
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3212:
URL: https://github.com/apache/bookkeeper/pull/3212#issuecomment-1101042990
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org