You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/12/18 00:46:39 UTC
[GitHub] [bookkeeper] dlg99 commented on a change in pull request #2936: BP-46: Data integrity check for running without journal
dlg99 commented on a change in pull request #2936:
URL: https://github.com/apache/bookkeeper/pull/2936#discussion_r771743310
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/UncleanShutdownDetectionImpl.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.bookkeeper.bookie;
+
+import java.io.File;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Used to determine if the prior shutdown was unclean or not. It does so
+ * by adding a file to each ledger directory after successful start-up
+ * and removing the file on graceful shutdown.
+ * Any abrupt termination will cause one or more of these files to not be cleared
+ * and so on the subsequent boot-up, the presence of any of these files will
+ * indicate an unclean shutdown.
+ */
+public class UncleanShutdownDetectionImpl implements UncleanShutdownDetection {
+ private static final Logger LOG = LoggerFactory.getLogger(UncleanShutdownDetectionImpl.class);
+ private final LedgerDirsManager ledgerDirsManager;
+ static final String DirtyFileName = "DIRTY";
+
+ public UncleanShutdownDetectionImpl(LedgerDirsManager ledgerDirsManager) {
+ this.ledgerDirsManager = ledgerDirsManager;
+ }
+
+ @Override
+ public void registerStartUp() {
+ for (File ledgerDir : ledgerDirsManager.getAllLedgerDirs()) {
+ try {
+ File dirtyFile = new File(ledgerDir, DirtyFileName);
+ dirtyFile.createNewFile();
+ } catch (Throwable t) {
+ LOG.error("Unable to register start-up (so an unclean shutdown cannot"
Review comment:
throw/crash in this case?
Otherwise one can get bookie started in an unclear state without data correctness guarantees later
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/datainteg/DataIntegrityCheck.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.bookkeeper.bookie.datainteg;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * The interface for the data integrity check feature. This feature allows
+ * a bookie to handle data loss scenarios such as when running without
+ * the journal or after a disk failure has caused the loss of all data.
+ */
+public interface DataIntegrityCheck {
+ /**
+ * Run quick preboot check. This check should do enough to ensure that
+ * it is safe to complete the boot sequence without compromising correctness.
+ * To this end, if it finds that this bookie is part of the last ensemble of
+ * an unclosed ledger, it must prevent the bookie from being able store new
+ * entries for that ledger and must prevent the bookie from taking part in
+ * the discovery of the last entry of that ledger.
+ */
+ CompletableFuture<Void> runPreBoot(String reason);
+
+ /**
+ * Whether we need to run a full check.
+ * This condition can be set by the runPreBoot() call to run a full check
+ * in the background once the bookie is running. This can later be used
+ * to run the full check periodically, or to exponentially backoff and retry
+ * when some transient condition prevents a ledger being fixed during a
+ * full check.
+ */
+ boolean needsFull() throws IOException;
+
+ /**
+ * Run full check of bookies local data. This check should ensure that
+ * if the metadata service states that it should have an entry, then it
+ * should have that entry. If the entry is missing, it should copy it
+ * from another available source.
+ */
+ CompletableFuture<Void> runFullCheck();
Review comment:
nit: naming consistency.
either runPreBoot/needsFull/runFull or runPreBootCheck/needsFullCheck/runFullCheck (I prefer the latter)
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtocol.java
##########
@@ -180,6 +180,11 @@ public static short getFlags(int packetHeader) {
*/
int ETOOMANYREQUESTS = 106;
+ /**
+ * Ledger in unknown state.
+ */
+ int EUNKNOWN = 107;
Review comment:
do we need to bump up CURRENT_PROTOCOL_VERSION?
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java
##########
@@ -1013,4 +1077,91 @@ public void diskJustWritable(File disk) {
}
};
}
+
+ @Override
+ public void setLimboState(long ledgerId) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("setLimboState. ledger: {}", ledgerId);
+ }
+ ledgerIndex.setLimbo(ledgerId);
+ }
+
+ @Override
+ public boolean hasLimboState(long ledgerId) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("hasLimboState. ledger: {}", ledgerId);
+ }
+ return ledgerIndex.get(ledgerId).getLimbo();
+ }
+
+ @Override
+ public void clearLimboState(long ledgerId) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("clearLimboState. ledger: {}", ledgerId);
+ }
+ ledgerIndex.clearLimbo(ledgerId);
+ }
+
+ private void throwIfLimbo(long ledgerId) throws IOException, BookieException {
Review comment:
maybe do `throws .. DataUnknownException` (instead of BookieException)? with similar changes wherever it bubbles up (readLastAddConfirmed/...), to make sure that some other Bookie Exception is not handled as a DataUnknown case.
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/datainteg/DataIntegrityCheckImpl.java
##########
@@ -0,0 +1,554 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.bookkeeper.bookie.datainteg;
+
+import com.google.common.collect.ImmutableSortedMap;
+import io.reactivex.rxjava3.core.Flowable;
+import io.reactivex.rxjava3.core.Maybe;
+import io.reactivex.rxjava3.core.Scheduler;
+import io.reactivex.rxjava3.core.Single;
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.bookie.BookieException;
+import org.apache.bookkeeper.bookie.LedgerStorage;
+import org.apache.bookkeeper.bookie.LedgerStorage.StorageState;
+import org.apache.bookkeeper.client.BKException;
+import org.apache.bookkeeper.client.BookKeeperAdmin;
+import org.apache.bookkeeper.client.api.LedgerMetadata;
+import org.apache.bookkeeper.common.concurrent.FutureUtils;
+import org.apache.bookkeeper.meta.LedgerManager;
+import org.apache.bookkeeper.net.BookieId;
+
+/**
+ * An implementation of the DataIntegrityCheck interface.
+ */
+@Slf4j
+public class DataIntegrityCheckImpl implements DataIntegrityCheck {
+ private static final int MAX_INFLIGHT = 300;
+ private static final int MAX_ENTRIES_INFLIGHT = 3000;
+ private static final int ZK_TIMEOUT_S = 30;
+ private final BookieId bookieId;
+ private final LedgerManager ledgerManager;
+ private final LedgerStorage ledgerStorage;
+ private final EntryCopier entryCopier;
+ private final BookKeeperAdmin admin;
+ private final Scheduler scheduler;
+ private final AtomicReference<Map<Long, LedgerMetadata>> ledgersCacheRef =
+ new AtomicReference<>(null);
+ private CompletableFuture<Void> preBootFuture;
+
+ public DataIntegrityCheckImpl(BookieId bookieId,
+ LedgerManager ledgerManager,
+ LedgerStorage ledgerStorage,
+ EntryCopier entryCopier,
+ BookKeeperAdmin admin,
+ Scheduler scheduler) {
+ this.bookieId = bookieId;
+ this.ledgerManager = ledgerManager;
+ this.ledgerStorage = ledgerStorage;
+ this.entryCopier = entryCopier;
+ this.admin = admin;
+ this.scheduler = scheduler;
+ }
+
+ @Override
+ public CompletableFuture<Void> runPreBoot(String reason) {
+ // we only run this once, it could be kicked off by different checks
+ synchronized (this) {
+ if (preBootFuture == null) {
+ preBootFuture = runPreBootSequence(reason);
+ }
+ }
+ return preBootFuture;
+
+ }
+
+ private CompletableFuture<Void> runPreBootSequence(String reason) {
+ String runId = UUID.randomUUID().toString();
+ log.info("Event: {}, RunId: {}, Reason: {}", Events.PREBOOT_START, runId, reason);
+ try {
+ this.ledgerStorage.setStorageStateFlag(StorageState.NEEDS_INTEGRITY_CHECK);
+ } catch (IOException ioe) {
+ log.error("Event: {}, RunId: {}", Events.PREBOOT_ERROR, runId, ioe);
+ return FutureUtils.exception(ioe);
+ }
+
+ MetadataAsyncIterator iter = new MetadataAsyncIterator(scheduler,
+ ledgerManager, MAX_INFLIGHT, ZK_TIMEOUT_S, TimeUnit.SECONDS);
+ CompletableFuture<Void> promise = new CompletableFuture<>();
+ Map<Long, LedgerMetadata> ledgersCache =
Review comment:
should we be concerned about memory requirements here in case of large disks (to many ledgers/metadata with many segments)? See https://github.com/apache/bookkeeper/pull/1949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org