You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by al...@apache.org on 2026/04/24 07:26:51 UTC
(kudu) branch master updated: [fs] add metrics for untracked orphaned blocks
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 80873c98b [fs] add metrics for untracked orphaned blocks
80873c98b is described below
commit 80873c98b5cc5110e611dd6180cfac12549e84f6
Author: Ashwani <ar...@gmail.com>
AuthorDate: Thu Mar 12 16:38:08 2026 +0530
[fs] add metrics for untracked orphaned blocks
During flush, if orphaned blocks are there and for some reason those
blocks' records are not deleted (maybe due to some underlying transient
I/O error, etc), those blocks are never re-tried for deletion and erased
from the list of orphaned_blocks_. This can cause un-reclaimed space on
the persistent storage that can get accumulated over time to a big size.
This un-reclaimed space can only be reclaimed with user intervention by
running 'kudu fs check --repair'.
Initially, this patch addressed the issue by not removing those orphaned
blocks from the orphaned_blocks_ list, for which delete failed due to
some transient error. This ensures that when next flush is scheduled,
the metadata still contains those orphaned blocks in the list and delete
can be retried on those blocks to reclaim the space.
However, with that change, TestEIODuringDelete started failing because
DeleteTabletData expects an empty orphaned_blocks_ set after calling
Flush() couple of times. A second flush is required to clear the
orphaned blocks from superblock even if delete failed for those earlier.
Since we keep adding the orphaned blocks (for which delete failed)
back to the set, it never got an empty set, that caused the failure.
If orphaned_blocks_ set is cleared after the second flush, to maintain
original behavior and expectation, it may fix the TestEIODuringDelete
failure but it opens the possibility of hitting KUDU-1060 where a number
tombstoned tablets keep the record of all orphaned blocks on persistent
superblock i.e., roll-forward of the block deletions until next restart.
This can severely impact the startup time for a tablet server that has
a lot of tombstoned tablets.
In a nutshell, having to keep these orphaned blocks in the set may not
be of much use if the disk error is persistent and not to mention the
additional handling required in various cases like mentioned above. It
makes sense to just rely on 'kudu fs check --repair' like workflow to
remove these stale orphaned blocks as a maintenance operation.
With all this information, re-purposing this patch to only focus on
adding additional logs, metrics and stats that can help identify the
scenarios with stale orphaned block id lists and log the appropriate
action for user i.e. 'kudu fs check --repair'. Original behavior holds
for orphaned blocks set i.e. erase all blocks from the set irrespective
of the CommitDeletedBlocks() outcome.
Follow-up:
KUDU-829 - Create a separate patch for adding a maintenance op that can
run in the background to reclaim all the space left from those orphaned
blocks using the same logic as 'kudu fs check --repair'.
Highlights of the change:
- Add warning logs at CommitDeletedBlocks callers when the commit fails.
- Add per-block level error logs for blocks for which deletion record
could not be committed.
- Add these metrics that can hold the orphan block deletion outcome:
* orphaned_blocks_cleaned
* orphaned_block_cleanup_failures
- Add unit tests to test these scenarios with metrics verification:
* Usual path where orphaned blocks are deleted with no error.
* No metrics are updated when orphan block deletion is disabled.
* Induced I/O error causes orphaned blocks lying around that are not
deleted and eventually removed from the set with a action to user.
* Two stages: First induce I/O error, verify metrics shows cleanup
failure count increased. Second remove induced error, verify metrics
remains unchanged.
Change-Id: Id386d9fc8d0900839e229e66772f35299b3ef2e9
Reviewed-on: http://gerrit.cloudera.org:8080/24092
Tested-by: Kudu Jenkins
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
Reviewed-by: Alexey Serbin <al...@apache.org>
---
src/kudu/fs/log_block_manager.cc | 31 ++++-
src/kudu/tablet/tablet.cc | 49 ++++++--
src/kudu/tablet/tablet.h | 7 +-
src/kudu/tablet/tablet_metadata-test.cc | 209 ++++++++++++++++++++++++++++++++
src/kudu/tablet/tablet_metadata.cc | 49 ++++++--
src/kudu/tablet/tablet_metadata.h | 22 +++-
src/kudu/tablet/tablet_metrics.cc | 18 +++
src/kudu/tablet/tablet_metrics.h | 6 +
8 files changed, 362 insertions(+), 29 deletions(-)
diff --git a/src/kudu/fs/log_block_manager.cc b/src/kudu/fs/log_block_manager.cc
index 98558a116..6b906e272 100644
--- a/src/kudu/fs/log_block_manager.cc
+++ b/src/kudu/fs/log_block_manager.cc
@@ -3358,6 +3358,7 @@ Status LogBlockManager::RemoveLogBlocks(const vector<BlockId>& block_ids,
lbs_by_container.emplace_back(std::move(lb));
}
+ size_t total_orphaned_count = 0;
for (auto& [container, clbs] : lbs_by_containers) {
for (const auto& lb : clbs) {
VLOG(3) << "Deleting block " << lb->block_id();
@@ -3376,14 +3377,33 @@ Status LogBlockManager::RemoveLogBlocks(const vector<BlockId>& block_ids,
// fsync).
//
// TODO(KUDU-829): Implement GC of orphaned blocks.
- // TODO(yingchun): Add some metrics to track the number of orphaned blocks.
if (s.ok()) {
container->PostWorkOfBlocksDeleted();
} else {
if (first_failure.ok()) {
first_failure = s.CloneAndPrepend("Unable to append deletion record(s) to block metadata");
}
- // Purge the blocks that failed to delete.
+ // Blocks whose deletion metadata could not be written are permanently
+ // orphaned i.e., their data occupies space on disk. They will not be cleaned
+ // up automatically and may cause unreclaimed disk space accumulation over time.
+ // RemoveBlockIdsFromMetadata writes records in the same order as clbs,
+ // so deleted_block_ids.size() is the index of the first failure: the
+ // failed blocks form a contiguous suffix of clbs.
+ const size_t orphaned_count =
+ clbs.size() - deleted_block_ids.size();
+ for (auto it = clbs.cbegin() + deleted_block_ids.size(); it != clbs.cend(); ++it) {
+ KLOG_EVERY_N_SECS(ERROR, 1) << Substitute(
+ "Block $0 in container $1 is now orphaned (failed to commit deletion record): $2, "
+ "with unreclaimed disk space left behind. Run 'kudu fs check --repair' to reclaim it.",
+ (*it)->block_id().ToString(), container->ToString(), s.ToString()) << THROTTLE_MSG;
+ }
+ KLOG_EVERY_N_SECS(ERROR, 1) << Substitute(
+ "$0 block(s) in container $1 are now orphaned (failed to commit "
+ "deletion records): $2, with unreclaimed disk space left behind. "
+ "Run 'kudu fs check --repair' to reclaim it.",
+ orphaned_count, container->ToString(), s.ToString()) << THROTTLE_MSG;
+ total_orphaned_count += orphaned_count;
+ // Purge the blocks that failed to delete from the current batch.
clbs.resize(deleted_block_ids.size());
}
@@ -3393,6 +3413,13 @@ Status LogBlockManager::RemoveLogBlocks(const vector<BlockId>& block_ids,
std::move(clbs.begin(), clbs.end(), std::back_inserter(*log_blocks));
}
+ if (total_orphaned_count > 0) {
+ LOG(ERROR) << Substitute(
+ "$0 block(s) across $1 container(s) are now orphaned with unreclaimed "
+ "disk space. Run 'kudu fs check --repair' to reclaim it.",
+ total_orphaned_count, lbs_by_containers.size());
+ }
+
return first_failure;
}
diff --git a/src/kudu/tablet/tablet.cc b/src/kudu/tablet/tablet.cc
index 58517984c..5e908dd4b 100644
--- a/src/kudu/tablet/tablet.cc
+++ b/src/kudu/tablet/tablet.cc
@@ -61,6 +61,7 @@
#include "kudu/fs/io_context.h"
#include "kudu/gutil/casts.h"
#include "kudu/gutil/map-util.h"
+#include "kudu/gutil/port.h"
#include "kudu/gutil/strings/human_readable.h"
#include "kudu/gutil/strings/substitute.h"
#include "kudu/gutil/threading/thread_collision_warner.h"
@@ -1738,7 +1739,7 @@ Status Tablet::AlterSchema(AlterSchemaOpState* op_state) {
// If the current schema and the new one are equal, there is nothing to do.
if (same_schema) {
- return metadata_->Flush();
+ return FlushTabletMetadataAndUpdateMetrics();
}
return FlushUnlocked();
@@ -1978,10 +1979,20 @@ void Tablet::CancelMaintenanceOps() {
}
}
+void Tablet::UpdateOrphanBlockMetrics(
+ const TabletMetadata::OrphanBlockCleanupStats& stats) {
+ if (!metrics_) {
+ return;
+ }
+ metrics_->orphaned_blocks_cleaned->IncrementBy(stats.blocks_cleaned);
+ metrics_->orphaned_block_cleanup_failures->IncrementBy(stats.blocks_failed);
+}
+
Status Tablet::FlushMetadata(const RowSetVector& to_remove,
const RowSetMetadataVector& to_add,
int64_t mrs_being_flushed,
const vector<TxnInfoBeingFlushed>& txns_being_flushed) {
+ TabletMetadata::OrphanBlockCleanupStats orphan_stats;
RowSetMetadataIds to_remove_meta;
for (const shared_ptr<RowSet>& rowset : to_remove) {
// Skip MemRowSet & DuplicatingRowSets which don't have metadata.
@@ -1991,8 +2002,18 @@ Status Tablet::FlushMetadata(const RowSetVector& to_remove,
to_remove_meta.insert(rowset->metadata()->id());
}
- return metadata_->UpdateAndFlush(to_remove_meta, to_add, mrs_being_flushed,
- txns_being_flushed);
+ RETURN_NOT_OK(metadata_->UpdateAndFlush(to_remove_meta, to_add, mrs_being_flushed,
+ txns_being_flushed, &orphan_stats));
+
+ UpdateOrphanBlockMetrics(orphan_stats);
+ return Status::OK();
+}
+
+Status Tablet::FlushTabletMetadataAndUpdateMetrics() {
+ TabletMetadata::OrphanBlockCleanupStats orphan_stats;
+ RETURN_NOT_OK(metadata_->Flush(&orphan_stats));
+ UpdateOrphanBlockMetrics(orphan_stats);
+ return Status::OK();
}
// Computes on-disk size of all the deltas in provided rowsets.
@@ -2250,9 +2271,11 @@ Status Tablet::DoMergeCompactionOrFlush(const RowSetsInCompactionOrFlush &input,
}
// Write out the new Tablet Metadata and remove old rowsets.
- RETURN_NOT_OK_PREPEND(FlushMetadata(input.rowsets(), new_drs_metas, mrs_being_flushed,
- txns_being_flushed),
- "Failed to flush new tablet metadata");
+ {
+ RETURN_NOT_OK_PREPEND(FlushMetadata(input.rowsets(), new_drs_metas, mrs_being_flushed,
+ txns_being_flushed),
+ "Failed to flush new tablet metadata");
+ }
// Now that we've completed the operation, mark any rowsets that have been
// compacted, preventing them from being considered for future compactions.
@@ -2316,11 +2339,13 @@ Status Tablet::HandleEmptyCompactionOrFlush(const RowSetVector& rowsets,
int mrs_being_flushed,
const vector<TxnInfoBeingFlushed>& txns_being_flushed) {
// Write out the new Tablet Metadata and remove old rowsets.
- RETURN_NOT_OK_PREPEND(FlushMetadata(rowsets,
- RowSetMetadataVector(),
- mrs_being_flushed,
- txns_being_flushed),
- "Failed to flush new tablet metadata");
+ {
+ RETURN_NOT_OK_PREPEND(FlushMetadata(rowsets,
+ RowSetMetadataVector(),
+ mrs_being_flushed,
+ txns_being_flushed),
+ "Failed to flush new tablet metadata");
+ }
AtomicSwapRowSets(rowsets, RowSetVector());
UpdateAverageRowsetHeight();
@@ -3152,7 +3177,7 @@ Status Tablet::DeleteAncientUndoDeltas(int64_t* blocks_deleted, int64_t* bytes_d
// We flush the tablet metadata at the end because we don't flush per-RowSet
// for performance reasons.
if (tablet_blocks_deleted > 0) {
- RETURN_NOT_OK(metadata_->Flush());
+ RETURN_NOT_OK(FlushTabletMetadataAndUpdateMetrics());
}
MonoDelta tablet_delete_duration = MonoTime::Now() - tablet_delete_start;
diff --git a/src/kudu/tablet/tablet.h b/src/kudu/tablet/tablet.h
index 8b4cd63ff..5ea760a25 100644
--- a/src/kudu/tablet/tablet.h
+++ b/src/kudu/tablet/tablet.h
@@ -39,7 +39,6 @@
#include "kudu/fs/io_context.h"
#include "kudu/gutil/integral_types.h"
#include "kudu/gutil/macros.h"
-#include "kudu/gutil/port.h"
#include "kudu/gutil/ref_counted.h"
#include "kudu/gutil/threading/thread_collision_warner.h"
#include "kudu/tablet/lock_manager.h"
@@ -761,6 +760,12 @@ class Tablet {
int64_t mrs_being_flushed,
const std::vector<TxnInfoBeingFlushed>& txns_being_flushed);
+ // Updates tablet metrics with the orphaned block info from metadata flush outcome.
+ void UpdateOrphanBlockMetrics(const TabletMetadata::OrphanBlockCleanupStats& orphan_stats);
+
+ // Flushes tablet metadata and updates metrics.
+ Status FlushTabletMetadataAndUpdateMetrics();
+
// Computes on-disk size of all the deltas in provided rowsets.
size_t GetAllDeltasSizeOnDisk(const RowSetsInCompactionOrFlush& input);
diff --git a/src/kudu/tablet/tablet_metadata-test.cc b/src/kudu/tablet/tablet_metadata-test.cc
index 72244da8e..f7ffcf332 100644
--- a/src/kudu/tablet/tablet_metadata-test.cc
+++ b/src/kudu/tablet/tablet_metadata-test.cc
@@ -27,6 +27,7 @@
#include <type_traits>
#include <unordered_map>
#include <unordered_set>
+#include <vector>
#include <gflags/gflags.h>
#include <glog/logging.h>
@@ -39,6 +40,8 @@
#include "kudu/common/wire_protocol-test-util.h"
#include "kudu/consensus/log_anchor_registry.h"
#include "kudu/fs/block_id.h"
+#include "kudu/fs/block_manager.h"
+#include "kudu/fs/fs_manager.h"
#include "kudu/gutil/map-util.h"
#include "kudu/gutil/ref_counted.h"
#include "kudu/tablet/local_tablet_writer.h"
@@ -46,21 +49,35 @@
#include "kudu/tablet/rowset_metadata.h"
#include "kudu/tablet/tablet-harness.h"
#include "kudu/tablet/tablet-test-util.h"
+#include "kudu/tablet/tablet_metrics.h"
#include "kudu/tablet/txn_metadata.h"
+#include "kudu/util/logging_test_util.h"
+#include "kudu/util/metrics.h"
#include "kudu/util/pb_util.h"
+#include "kudu/util/slice.h"
#include "kudu/util/status.h"
#include "kudu/util/stopwatch.h"
#include "kudu/util/test_macros.h"
+DECLARE_bool(enable_tablet_orphaned_block_deletion);
+DECLARE_double(env_inject_eio);
+DECLARE_string(env_inject_eio_globs);
+
DEFINE_int64(test_row_set_count, 1000, "");
DEFINE_int64(test_block_count_per_rs, 1000, "");
+using kudu::fs::BlockManager;
+using kudu::fs::CreateBlockOptions;
+using kudu::fs::ReadableBlock;
+using kudu::fs::WritableBlock;
using kudu::log::LogAnchorRegistry;
using kudu::log::MinLogIndexAnchorer;
using std::map;
using std::shared_ptr;
+using std::string;
using std::unique_ptr;
using std::unordered_set;
+using std::vector;
namespace kudu {
namespace tablet {
@@ -80,6 +97,11 @@ class TestTabletMetadata : public KuduTabletTest {
void BuildPartialRow(int key, int intval, const char* strval,
unique_ptr<KuduPartialRow>* row);
+ // Creates a block on disk using the block manager, writes some data to it, and returns
+ // its BlockId. The block is committed to the block manager's in-memory index and
+ // can be added to the tablet metadata's orphaned-blocks set for deletion testing.
+ BlockId CreateRealBlock();
+
protected:
unique_ptr<LocalTabletWriter> writer_;
};
@@ -92,6 +114,17 @@ void TestTabletMetadata::BuildPartialRow(int key, int intval, const char* strval
CHECK_OK((*row)->SetStringCopy(2, strval));
}
+BlockId TestTabletMetadata::CreateRealBlock() {
+ scoped_refptr<BlockManager> bm = harness_->fs_manager()->block_manager();
+ const string& tablet_id = harness_->tablet()->metadata()->tablet_id();
+ unique_ptr<WritableBlock> block;
+ CHECK_OK(bm->CreateBlock(CreateBlockOptions{tablet_id}, &block));
+ BlockId block_id = block->id();
+ CHECK_OK(block->Append(Slice("orphaned block test data")));
+ CHECK_OK(block->Close());
+ return block_id;
+}
+
// Test that loading & storing the superblock results in an equivalent file.
TEST_F(TestTabletMetadata, TestLoadFromSuperBlock) {
TabletMetadata* meta = harness_->tablet()->metadata();
@@ -331,5 +364,181 @@ TEST_F(TestTabletMetadata, TestTxnMetadata) {
NO_FATALS(validate_txn_metas(new_meta.get()));
}
+// Verify the happy path i.e., orphaned blocks are successfully deleted during a
+// metadata flush, and the tablet metrics are updated to reflect the cleanup.
+TEST_F(TestTabletMetadata, TestOrphanBlockCleanupSuccess) {
+ TabletMetadata* meta = harness_->tablet()->metadata();
+
+ // Create a real block on disk and register it as orphaned.
+ BlockId block_id = CreateRealBlock();
+ meta->AddOrphanedBlocks({block_id});
+
+ // Flush the tablet (the MemRowSet is empty, so HandleEmptyCompactionOrFlush
+ // is taken, which still runs the full metadata flush and orphan cleanup path).
+ StringVectorSink sink;
+ {
+ ScopedRegisterSink reg(&sink);
+ ASSERT_OK(harness_->tablet()->Flush());
+ }
+
+ // Metrics must reflect one successful cleanup and zero failures.
+ auto* metrics = harness_->tablet()->metrics();
+ ASSERT_EQ(1, metrics->orphaned_blocks_cleaned->value());
+ ASSERT_EQ(0, metrics->orphaned_block_cleanup_failures->value());
+
+ // No error or warning about orphaned-block cleanup should have been logged.
+ for (const string& s : sink.logged_msgs()) {
+ ASSERT_STR_NOT_MATCHES(s, ": failed cleaning up.*removed block");
+ ASSERT_STR_NOT_MATCHES(s, "Not deleting.*orphaned block");
+ }
+
+ // The block must no longer be accessible via the block manager.
+ unique_ptr<ReadableBlock> rblock;
+ Status s = harness_->fs_manager()->block_manager()->OpenBlock(block_id, &rblock);
+ ASSERT_TRUE(s.IsNotFound()) << "Expected block to be deleted: " << s.ToString();
+}
+
+// Verify that when deletion of orphaned block is disabled via the runtime flag, a
+// warning is logged, the block is not deleted, and no failure metrics are updated.
+TEST_F(TestTabletMetadata, TestOrphanBlockDeletionDisabled) {
+ TabletMetadata* meta = harness_->tablet()->metadata();
+
+ BlockId block_id = CreateRealBlock();
+ meta->AddOrphanedBlocks({block_id});
+
+ StringVectorSink sink;
+ {
+ // Disable orphaned block deletion for the duration of this scope.
+ google::FlagSaver flag_saver;
+ FLAGS_enable_tablet_orphaned_block_deletion = false;
+
+ ScopedRegisterSink reg(&sink);
+ ASSERT_OK(harness_->tablet()->Flush());
+ }
+
+ // A WARNING must have been emitted explaining that deletion was skipped.
+ ASSERT_STRINGS_ANY_MATCH(sink.logged_msgs(),
+ "Not deleting.*orphaned block.*enable_tablet_orphaned_block_deletion");
+
+ // Metrics must be all-zero: no deletion was attempted.
+ auto* metrics = harness_->tablet()->metrics();
+ ASSERT_EQ(0, metrics->orphaned_blocks_cleaned->value());
+ ASSERT_EQ(0, metrics->orphaned_block_cleanup_failures->value());
+
+ // The block must still be accessible on disk since deletion was suppressed.
+ unique_ptr<ReadableBlock> rblock;
+ ASSERT_OK(harness_->fs_manager()->block_manager()->OpenBlock(block_id, &rblock));
+}
+
+// Verify the failure path: when a block deletion encounters an I/O error
+// (simulated by injecting EIO on the data directory), an WARNING is logged
+// with the "kudu fs check --repair" hint and the failure metric is bumped.
+// Even though the physical deletion failed, the block is removed from
+// orphaned_blocks_ so it does not accumulate in the superblock across restarts.
+// The leaked disk space can be reclaimed with 'kudu fs check --repair'.
+TEST_F(TestTabletMetadata, TestOrphanBlockDeletionFailed) {
+ TabletMetadata* meta = harness_->tablet()->metadata();
+
+ BlockId block_id = CreateRealBlock();
+ meta->AddOrphanedBlocks({block_id});
+
+ // Identify the data directory so that we can scope EIO injection to it,
+ // leaving the tablet-metadata directory (superblock write path) unaffected.
+ const vector<string> data_dirs = harness_->fs_manager()->GetDataRootDirs();
+ ASSERT_FALSE(data_dirs.empty());
+
+ StringVectorSink sink;
+ {
+ // Inject EIO on every file operation under the data directory. The
+ // superblock write goes to tablet-meta which is a different directory and
+ // therefore unaffected. Also, Tablet::Flush() still returns OK overall.
+ google::FlagSaver flag_saver;
+ FLAGS_env_inject_eio_globs = data_dirs[0] + "/*";
+ FLAGS_env_inject_eio = 1.0;
+
+ ScopedRegisterSink reg(&sink);
+ // Flush succeeds overall; orphan deletion failure is non-fatal.
+ ASSERT_OK(harness_->tablet()->Flush());
+ }
+
+ // An warning log must report the failed cleanup and suggest the repair tool.
+ ASSERT_STRINGS_ANY_MATCH(sink.logged_msgs(),
+ ": failed cleaning up.*removed block");
+ ASSERT_STRINGS_ANY_MATCH(sink.logged_msgs(),
+ "kudu fs check --repair");
+
+ // The failure counter must be incremented; cleaned must remain zero.
+ auto* metrics = harness_->tablet()->metrics();
+ ASSERT_EQ(0, metrics->orphaned_blocks_cleaned->value());
+ ASSERT_EQ(1, metrics->orphaned_block_cleanup_failures->value());
+
+ // Even though physical deletion failed, the block must have been removed
+ // from orphaned_blocks_ so the superblock is written clean. This prevents
+ // the entry from accumulating across restarts. The block remains on disk
+ // as leaked space, recoverable via 'kudu fs check --repair'.
+ TabletSuperBlockPB pb;
+ ASSERT_OK(meta->ToSuperBlock(&pb));
+ ASSERT_EQ(0, pb.orphaned_blocks_size())
+ << "Block must be removed from orphaned set even on deletion failure";
+}
+
+// Verify that a block whose physical deletion fails due to an I/O error is
+// not retained in orphaned_blocks_ for retry. After the flush the superblock
+// is written clean (zero orphaned entries) and a subsequent flush performs no
+// additional cleanup work. The leaked block remains on disk and is recoverable
+// via 'kudu fs check --repair'.
+TEST_F(TestTabletMetadata, TestOrphanBlockNotRetainedAfterFailure) {
+ TabletMetadata* meta = harness_->tablet()->metadata();
+
+ BlockId block_id = CreateRealBlock();
+ meta->AddOrphanedBlocks({block_id});
+
+ const vector<string> data_dirs = harness_->fs_manager()->GetDataRootDirs();
+ ASSERT_FALSE(data_dirs.empty());
+
+ StringVectorSink sink;
+
+ // First flush: deletion fails due to injected EIO.
+ {
+ ScopedRegisterSink reg(&sink);
+ google::FlagSaver flag_saver;
+ FLAGS_env_inject_eio_globs = data_dirs[0] + "/*";
+ FLAGS_env_inject_eio = 1.0;
+ // Flush still returns OK; orphan deletion failure is non-fatal.
+ ASSERT_OK(harness_->tablet()->Flush());
+ }
+
+ ASSERT_STRINGS_ANY_MATCH(sink.logged_msgs(),
+ ": failed cleaning up.*removed block");
+
+ auto* metrics = harness_->tablet()->metrics();
+ ASSERT_EQ(0, metrics->orphaned_blocks_cleaned->value());
+ ASSERT_EQ(1, metrics->orphaned_block_cleanup_failures->value());
+
+ // The block must have been removed from orphaned_blocks_ even though its
+ // physical deletion failed. The superblock must already be clean.
+ TabletSuperBlockPB pb;
+ ASSERT_OK(meta->ToSuperBlock(&pb));
+ ASSERT_EQ(0, pb.orphaned_blocks_size())
+ << "Block must be removed from orphaned set even on deletion failure";
+
+ // Second flush: no I/O error, but there is nothing left to retry.
+ sink.logged_msgs().clear();
+ {
+ ScopedRegisterSink reg(&sink);
+ ASSERT_OK(harness_->tablet()->Flush());
+ }
+
+ // No orphan-related messages should appear — there was nothing to clean up.
+ for (const string& s : sink.logged_msgs()) {
+ ASSERT_STR_NOT_MATCHES(s, ": failed cleaning up.*removed block");
+ ASSERT_STR_NOT_MATCHES(s, "Not deleting.*orphaned block");
+ }
+
+ // Counters must be unchanged: no additional cleanup was attempted.
+ ASSERT_EQ(0, metrics->orphaned_blocks_cleaned->value());
+ ASSERT_EQ(1, metrics->orphaned_block_cleanup_failures->value());
+}
+
} // namespace tablet
} // namespace kudu
diff --git a/src/kudu/tablet/tablet_metadata.cc b/src/kudu/tablet/tablet_metadata.cc
index 6ef326d7e..9816997a1 100644
--- a/src/kudu/tablet/tablet_metadata.cc
+++ b/src/kudu/tablet/tablet_metadata.cc
@@ -543,12 +543,13 @@ Status TabletMetadata::LoadFromSuperBlock(const TabletSuperBlockPB& superblock)
Status TabletMetadata::UpdateAndFlush(const RowSetMetadataIds& to_remove,
const RowSetMetadataVector& to_add,
int64_t last_durable_mrs_id,
- const vector<TxnInfoBeingFlushed>& txns_being_flushed) {
+ const vector<TxnInfoBeingFlushed>& txns_being_flushed,
+ OrphanBlockCleanupStats* orphan_stats) {
{
std::lock_guard l(data_lock_);
RETURN_NOT_OK(UpdateUnlocked(to_remove, to_add, last_durable_mrs_id, txns_being_flushed));
}
- return Flush();
+ return Flush(orphan_stats);
}
void TabletMetadata::AddOrphanedBlocks(const BlockIdContainer& block_ids) {
@@ -558,15 +559,23 @@ void TabletMetadata::AddOrphanedBlocks(const BlockIdContainer& block_ids) {
void TabletMetadata::AddOrphanedBlocksUnlocked(const BlockIdContainer& block_ids) {
DCHECK(data_lock_.is_locked());
+ VLOG_WITH_PREFIX(1) << Substitute("Scheduling $0 orphaned block(s) for cleanup, "
+ "total pending: $1", block_ids.size(),
+ orphaned_blocks_.size() + block_ids.size());
orphaned_blocks_.insert(block_ids.begin(), block_ids.end());
}
-void TabletMetadata::DeleteOrphanedBlocks(const BlockIdContainer& blocks) {
+TabletMetadata::OrphanBlockCleanupStats TabletMetadata::DeleteOrphanedBlocks(
+ const BlockIdContainer& blocks) {
+ if (blocks.empty()) {
+ return {};
+ }
+
if (PREDICT_FALSE(!FLAGS_enable_tablet_orphaned_block_deletion)) {
- LOG_WITH_PREFIX(WARNING) << "Not deleting " << blocks.size()
- << " block(s) from disk. Block deletion disabled via "
- << "--enable_tablet_orphaned_block_deletion=false";
- return;
+ LOG_WITH_PREFIX(WARNING) << Substitute(
+ "Not deleting $0 orphaned block(s) from disk. Block deletion disabled via "
+ "--enable_tablet_orphaned_block_deletion=false", blocks.size());
+ return {};
}
auto bm = fs_manager()->block_manager();
@@ -574,8 +583,22 @@ void TabletMetadata::DeleteOrphanedBlocks(const BlockIdContainer& blocks) {
for (const BlockId& b : blocks) {
transaction->AddDeletedBlock(b);
}
- WARN_NOT_OK(transaction->CommitDeletedBlocks(nullptr),
- "not all orphaned blocks were deleted");
+
+ vector<BlockId> deleted_blocks;
+ Status s = transaction->CommitDeletedBlocks(&deleted_blocks);
+
+ OrphanBlockCleanupStats stats;
+ stats.blocks_cleaned = static_cast<int64_t>(deleted_blocks.size());
+ stats.blocks_failed = static_cast<int64_t>(blocks.size()) - stats.blocks_cleaned;
+
+ if (PREDICT_FALSE(!s.ok())) {
+ LOG_WITH_PREFIX(WARNING) << Substitute(
+ "$2: failed cleaning up $0 of $1 removed block(s); running "
+ "'kudu fs check --repair' can help to reclaim the disk space.",
+ stats.blocks_failed, blocks.size(), s.ToString());
+ } else {
+ VLOG_WITH_PREFIX(1) << "Deleted " << stats.blocks_cleaned << " orphaned block(s)";
+ }
// Regardless of whether we deleted all the blocks or not, remove them from
// the orphaned blocks list. If we failed to delete the blocks due to
@@ -587,6 +610,7 @@ void TabletMetadata::DeleteOrphanedBlocks(const BlockIdContainer& blocks) {
orphaned_blocks_.erase(b);
}
}
+ return stats;
}
void TabletMetadata::PinFlush() {
@@ -607,7 +631,7 @@ Status TabletMetadata::UnPinFlush() {
return Status::OK();
}
-Status TabletMetadata::Flush() {
+Status TabletMetadata::Flush(OrphanBlockCleanupStats* orphan_stats) {
TRACE_EVENT1("tablet", "TabletMetadata::Flush",
"tablet_id", tablet_id_);
@@ -649,7 +673,10 @@ Status TabletMetadata::Flush() {
//
// If we crash just before the deletion, we'll retry when reloading from
// disk; the orphaned blocks were persisted as part of the superblock.
- DeleteOrphanedBlocks(orphaned);
+ OrphanBlockCleanupStats stats = DeleteOrphanedBlocks(orphaned);
+ if (orphan_stats) {
+ *orphan_stats = stats;
+ }
return Status::OK();
}
diff --git a/src/kudu/tablet/tablet_metadata.h b/src/kudu/tablet/tablet_metadata.h
index d87991a11..12516c477 100644
--- a/src/kudu/tablet/tablet_metadata.h
+++ b/src/kudu/tablet/tablet_metadata.h
@@ -185,7 +185,20 @@ class TabletMetadata : public RefCountedThreadSafe<TabletMetadata> {
// this method.
Status UnPinFlush();
- Status Flush();
+ // Stats produced by a orphaned block cleanup pass that runs at the end of each Flush() call.
+ struct OrphanBlockCleanupStats {
+ // Number of orphaned blocks successfully deleted from disk.
+ int64_t blocks_cleaned = 0;
+ // Number of orphaned blocks that could not be deleted (e.g. I/O error).
+ // These are removed from the superblock regardless (see DeleteOrphanedBlocks),
+ // so any unreclaimed disk space must be recovered via 'kudu fs check --repair'.
+ int64_t blocks_failed = 0;
+ };
+
+ // Flushes the tablet superblock to disk and then attempts to delete any
+ // orphaned blocks recorded in the superblock. If 'orphan_stats' is non-null
+ // it is populated with the outcome of the cleanup attempt.
+ Status Flush(OrphanBlockCleanupStats* orphan_stats = nullptr);
// Updates the metadata in the following ways:
// 1. Adds rowsets from 'to_add'.
@@ -193,11 +206,14 @@ class TabletMetadata : public RefCountedThreadSafe<TabletMetadata> {
// 3. Adds orphaned blocks from 'to_remove'.
// 4. Updates the last durable MRS ID from 'last_durable_mrs_id',
// assuming it's not kNoMrsFlushed.
+ // 5. If 'orphan_stats' is non-null it receives the orphaned block cleanup
+ // outcome (see Flush()).
static const int64_t kNoMrsFlushed = -1;
Status UpdateAndFlush(const RowSetMetadataIds& to_remove,
const RowSetMetadataVector& to_add,
int64_t last_durable_mrs_id,
- const std::vector<TxnInfoBeingFlushed>& txns_being_flushed = {});
+ const std::vector<TxnInfoBeingFlushed>& txns_being_flushed = {},
+ OrphanBlockCleanupStats* orphan_stats = nullptr);
// Adds the blocks referenced by 'block_ids' to 'orphaned_blocks_'.
//
@@ -404,7 +420,7 @@ class TabletMetadata : public RefCountedThreadSafe<TabletMetadata> {
// 'orphaned_blocks_' set.
//
// Failures are logged, but are not fatal.
- void DeleteOrphanedBlocks(const BlockIdContainer& blocks);
+ OrphanBlockCleanupStats DeleteOrphanedBlocks(const BlockIdContainer& blocks);
enum State {
kNotLoadedYet,
diff --git a/src/kudu/tablet/tablet_metrics.cc b/src/kudu/tablet/tablet_metrics.cc
index 3d5a216c7..096064e5e 100644
--- a/src/kudu/tablet/tablet_metrics.cc
+++ b/src/kudu/tablet/tablet_metrics.cc
@@ -218,6 +218,22 @@ METRIC_DEFINE_counter(tablet, ops_timed_out_in_prepare_queue,
"acknowledged with TimedOut error status.",
kudu::MetricLevel::kWarn);
+METRIC_DEFINE_counter(tablet, orphaned_blocks_cleaned,
+ "Orphaned Blocks Cleaned",
+ kudu::MetricUnit::kBlocks,
+ "Number of orphaned blocks successfully deleted from disk. "
+ "Orphaned blocks are created during compaction and rowset GC "
+ "and are cleaned up after each metadata flush.",
+ kudu::MetricLevel::kInfo);
+
+METRIC_DEFINE_counter(tablet, orphaned_block_cleanup_failures,
+ "Orphaned Block Cleanup Failures",
+ kudu::MetricUnit::kBlocks,
+ "Number of orphaned blocks that could not be deleted from disk "
+ "during a cleanup attempt. A non-zero value may indicate disk "
+ "I/O errors. Failed blocks are retried on the next metadata flush.",
+ kudu::MetricLevel::kWarn);
+
METRIC_DEFINE_histogram(tablet, bloom_lookups_per_op, "Bloom Lookups per Operation",
kudu::MetricUnit::kProbes,
"Tracks the number of bloom filter lookups performed by each "
@@ -472,6 +488,8 @@ TabletMetrics::TabletMetrics(const scoped_refptr<MetricEntity>& entity)
MINIT(deleted_rowset_gc_bytes_deleted),
MINIT(undo_delta_block_gc_bytes_deleted),
MINIT(ops_timed_out_in_prepare_queue),
+ MINIT(orphaned_blocks_cleaned),
+ MINIT(orphaned_block_cleanup_failures),
MINIT(bloom_lookups_per_op),
MINIT(key_file_lookups_per_op),
MINIT(delta_file_lookups_per_op),
diff --git a/src/kudu/tablet/tablet_metrics.h b/src/kudu/tablet/tablet_metrics.h
index 5c75ecb3a..4153c910a 100644
--- a/src/kudu/tablet/tablet_metrics.h
+++ b/src/kudu/tablet/tablet_metrics.h
@@ -81,6 +81,12 @@ struct TabletMetrics {
scoped_refptr<Counter> undo_delta_block_gc_bytes_deleted;
scoped_refptr<Counter> ops_timed_out_in_prepare_queue;
+ // Orphaned block stats.
+ // Number of orphaned blocks successfully deleted from disk.
+ scoped_refptr<Counter> orphaned_blocks_cleaned;
+ // Number of orphaned blocks that could not be deleted due to underlying error.
+ scoped_refptr<Counter> orphaned_block_cleanup_failures;
+
scoped_refptr<Histogram> bloom_lookups_per_op;
scoped_refptr<Histogram> key_file_lookups_per_op;
scoped_refptr<Histogram> delta_file_lookups_per_op;