You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by mc...@apache.org on 2021/10/16 12:58:50 UTC
[cassandra-website] branch trunk updated: Added September 2021 blog
post: "Reaper: Anti-entropy Repairs Made Easy" about running repairs using
Reaper
This is an automated email from the ASF dual-hosted git repository.
mck pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-website.git
The following commit(s) were added to refs/heads/trunk by this push:
new ebdfcf2 Added September 2021 blog post: "Reaper: Anti-entropy Repairs Made Easy" about running repairs using Reaper
ebdfcf2 is described below
commit ebdfcf2027eefeba75efda3cb61a481b49041e0f
Author: Diogenese Topper <di...@gmail.com>
AuthorDate: Tue Sep 14 15:08:54 2021 -0700
Added September 2021 blog post: "Reaper: Anti-entropy Repairs Made Easy" about running repairs using Reaper
patch by Alexander Dejanovski, Diogenese Topper; reviewed by Erick Ramirez, Mick Semb Wever for CASSANDRA-16964
---
.../image1.png | Bin 0 -> 122235 bytes
.../image2.png | Bin 0 -> 137881 bytes
.../image3.png | Bin 0 -> 128102 bytes
.../image4.png | Bin 0 -> 99198 bytes
.../image5.png | Bin 0 -> 74683 bytes
.../image6.png | Bin 0 -> 341337 bytes
.../image7.png | Bin 0 -> 119139 bytes
.../image8.png | Bin 0 -> 131648 bytes
site-content/source/modules/ROOT/pages/blog.adoc | 28 ++++++-
.../blog/Reaper-Anti-entropy-Repair-Made-Easy.adoc | 82 +++++++++++++++++++++
10 files changed, 108 insertions(+), 2 deletions(-)
diff --git a/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image1.png b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image1.png
new file mode 100644
index 0000000..0b34c65
Binary files /dev/null and b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image1.png differ
diff --git a/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image2.png b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image2.png
new file mode 100644
index 0000000..2b3faa3
Binary files /dev/null and b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image2.png differ
diff --git a/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image3.png b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image3.png
new file mode 100644
index 0000000..9d4e8e6
Binary files /dev/null and b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image3.png differ
diff --git a/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image4.png b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image4.png
new file mode 100644
index 0000000..a0b5b8a
Binary files /dev/null and b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image4.png differ
diff --git a/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image5.png b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image5.png
new file mode 100644
index 0000000..45b154c
Binary files /dev/null and b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image5.png differ
diff --git a/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image6.png b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image6.png
new file mode 100644
index 0000000..5666c53
Binary files /dev/null and b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image6.png differ
diff --git a/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image7.png b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image7.png
new file mode 100644
index 0000000..d79e560
Binary files /dev/null and b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image7.png differ
diff --git a/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image8.png b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image8.png
new file mode 100644
index 0000000..a9aa6b2
Binary files /dev/null and b/site-content/source/modules/ROOT/images/blog/blog-post-reaper-anti-entropy-repair/image8.png differ
diff --git a/site-content/source/modules/ROOT/pages/blog.adoc b/site-content/source/modules/ROOT/pages/blog.adoc
index e73eb44..e190232 100644
--- a/site-content/source/modules/ROOT/pages/blog.adoc
+++ b/site-content/source/modules/ROOT/pages/blog.adoc
@@ -39,9 +39,33 @@ xref:blog/Apache-Cassandra-Changelog-10-October-2021.adoc[Read More]
[openblock,card-header]
------
[discrete]
+=== Reaper: Anti-entropy Repair Made Easy
+[discrete]
+==== September 28, 2021
+------
+[openblock,card-content]
+------
+Originally designed by Spotify, Reaper is an open source written in Java to schedule and orchestrate repairs of Apache Cassandra clusters. It helps make repairs as safe and reliable as possible, and with the recent release of Apache Cassandra 4.0 that also includes incremental repairs.
+
+[openblock,card-btn card-btn--blog]
+--------
+[.btn.btn--alt]
+xref:blog/Reaper-Anti-entropy-Repair-Made-Easy.adoc[Read More]
+--------
+
+------
+----
+//end card
+
+//start card
+[openblock,card shadow relative test]
+----
+[openblock,card-header]
+------
+[discrete]
=== Join Cassandra at Apachecon 2021
[discrete]
-==== August 27, 2021
+==== September 20, 2021
------
[openblock,card-content]
------
@@ -66,7 +90,7 @@ xref:blog/Join-Cassandra-at-ApacheCon-2021.adoc[Read More]
[discrete]
=== Cassandra on Kubernetes: A Beginner's Guide
[discrete]
-==== August 27, 2021
+==== September 4, 2021
------
[openblock,card-content]
------
diff --git a/site-content/source/modules/ROOT/pages/blog/Reaper-Anti-entropy-Repair-Made-Easy.adoc b/site-content/source/modules/ROOT/pages/blog/Reaper-Anti-entropy-Repair-Made-Easy.adoc
new file mode 100644
index 0000000..ca14fb1
--- /dev/null
+++ b/site-content/source/modules/ROOT/pages/blog/Reaper-Anti-entropy-Repair-Made-Easy.adoc
@@ -0,0 +1,82 @@
+= Reaper: Anti-entropy Repair Made Easy
+:page-layout: single-post
+:page-role: blog-post
+:page-post-date: September 28, 2021
+:page-post-author: Alexander Dejanovski
+:description: The Apache Cassandra Community
+:keywords:
+
+Reaper is an open source tool written in Java and built with Dropwizard to schedule and orchestrate repairs of Apache Cassandra clusters. It was originally https://www.slideshare.net/planetcassandra/spotify-automating-cassandra-repairs[designed and open-sourced by Spotify,window=_blank] in an attempt to automate repairs while applying best practices from their solid production experience.
+
+=== Repair Challenges
+
+Anti-entropy repair is traditionally performed using the nodetool repair command. It can be performed in two ways, full or incremental, and configured to repair various token ranges: all, primary range, or sub-range. Add to this different validation compaction orchestration settings (sequential, parallel, and data center aware), the fact that anti-compaction may trigger in some cases, and you're down a rabbit hole of complexity. All this for an operation that is mandatory and should be s [...]
+In the 1.x/2.x days of Cassandra (and probably after that), some operators simply gave up on repairing their clusters due to the difficulties involved in completing the operation successfully without impacting SLAs.
+The main problems encountered during repairs were:
+
+* A high number of pending compactions and SSTables on disk
+* Repairs taking longer than the tombstones GC grace period
+* High cluster load due to repair pressure
+* Blocked/never-ending repairs
+* A repair that isn't resumable in case of failure
+* vnodes made the operation very long and challenging to perform
+
+==== Reaper performs safe repairs
+
+Reaper was built to address those issues and make repairs as safe and reliable as possible. It splits the repair operations into evenly sized subranges and schedules them so that:
+
+* All nodes are kept busy repairing small units of data if possible
+* A single segment is running on a node at once
+* Segments lasting too long are terminated and re-scheduled
+* Failed segments get replayed in case of a transient failure
+* Pending compactions are monitored to pause segment scheduling, preventing overload
+* Repairs can be paused
+
+The tool also supports incremental repair, which should be safely usable starting with Cassandra 4.0. Since Cassandra 3.0, Reaper can create segments with several token ranges to reduce the overhead of vnodes on repairs. Such ranges will be repaired in a single job by Cassandra as segments will only contain ranges that are replicated on the same set of nodes.
+
+=== Reaper Features
+
+Reaper ships with a http://cassandra-reaper.io/docs/api/[REST API,window=_blank], a command-line tool (spreaper) and a Web UI:
+
+image::blog/blog-post-reaper-anti-entropy-repair/image5.png[cluster view]
+
+It collects and displays runtime Cassandra metrics, running compactions and ongoing streaming sessions:
+
+image::blog/blog-post-reaper-anti-entropy-repair/image6.png[node metrics]
+
+Reaper ships with a scheduler for recurring repairs but can also perform on-demand one-off repairs:
+
+image::blog/blog-post-reaper-anti-entropy-repair/image7.png[scheduler]
+
+image::blog/blog-post-reaper-anti-entropy-repair/image4.png[repair]
+
+It's easy to install as a http://cassandra-reaper.io/docs/download/install/[tarball,window=_blank], a https://hub.docker.com/r/thelastpickle/cassandra-reaper/[Docker container,window=_blank], or https://cloudsmith.io/%7Ethelastpickle/repos/reaper/packages/[deb/rpm packages,window=_blank] and can be deployed in various ways to accommodate your http://cassandra-reaper.io/docs/usage/multi_dc_distributed/[cluster's architecture,window=_blank]. A single Reaper instance is capable of managing [...]
+
+image::blog/blog-post-reaper-anti-entropy-repair/image8.png[singlereaper multidc]
+
+image::blog/blog-post-reaper-anti-entropy-repair/image3.png[multireaper multidc]
+
+image::blog/blog-post-reaper-anti-entropy-repair/image2.png[ha reaper setup]
+
+If JMX is restricted to local access, Reaper can even be deployed as a sidecar:
+
+image::blog/blog-post-reaper-anti-entropy-repair/image1.png[reaper sidecar]
+
+Reaper also can listen and display live Cassandra’s emitted Diagnostic Events.
+
+In Cassandra 4.0 internal system “diagnostic events” have become available via the work done in https://issues.apache.org/jira/browse/CASSANDRA-12944[CASSANDRA-12944,window=_blank]. These allow us to observe internal Cassandra events, for example, in unit tests, and with external tools. These diagnostic events provide operational monitoring and troubleshooting beyond logs and metrics.
+
+Reaper can use Postgres and Cassandra itself as a storage backend for its data and can repair all Cassandra versions since 1.2 up to the latest 4.0.
+
+To make Reaper more efficient, segment orchestration was recently revamped and modernized. It opened for a long-awaited feature: fully concurrent repairs for different keyspaces and tables.
+These changes also introduced a long-awaited feature by allowing fully concurrent repairs for different keyspaces/tables.
+
+You can find more details on these changes in the https://thelastpickle.com/blog/2021/02/22/reaper-for-apache-cassandra-2-2-release.html[2.2 release blog post,window=_blank].
+
+Note: the latest release is https://github.com/thelastpickle/cassandra-reaper/releases/tag/2.3.1[2.3.1,window=_blank].
+
+=== Eager to try Reaper?
+
+Head over to the http://cassandra-reaper.io/[cassandra-reaper.io,window=_blank] website, which contains all information you'll need to get started, install Reaper and stop worrying about repairs!
+
+This article by Alexander Dejanovski was previously published on https://jaxenter.com/reaper-anti-entropy-174878.html[JAXEnter.com,window=_blank].
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org