You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Kenneth Brotman (JIRA)" <ji...@apache.org> on 2018/03/01 16:06:00 UTC
[jira] [Comment Edited] (CASSANDRA-14272) The Backups web page on
the web site is empty
[ https://issues.apache.org/jira/browse/CASSANDRA-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382124#comment-16382124 ]
Kenneth Brotman edited comment on CASSANDRA-14272 at 3/1/18 4:05 PM:
---------------------------------------------------------------------
From DataStax at [https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupRestoreTOC.html]
was (Author: kenbrotman):
From DataStax at [https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupRestoreTOC.html]
h1. Backing up and restoring data
DataStax OpsCenter provides automated backup and restore functionality, see [Backup Service|https://docs.datastax.com/en/opscenter/latest/opsc/online_help/services/opscBackupService.html].
* *[About snapshots|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAboutSnapshots.html]*
A brief description of how DataStax Enterprise backs up data.
* *[Taking a snapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupTakesSnapshot.html]*
Steps for taking a global snapshot or per node.
* *[Deleting snapshot files|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupDeleteSnapshot.html]*
Steps to delete snapshot files.
* *[Enabling incremental backups|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupIncremental.html]*
Steps to enable incremental backups. When incremental backups are enabled, DataStax Enterprise hard-links each memtable flushed to an SSTable to a backups directory under the keyspace data directory.
* *[Restoring from a snapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupSnapshotRestore.html]*
Methods for restoring from a snapshot.
* *[Restoring a snapshot into a new cluster|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html]*
Steps for restoring a snapshot by recovering the cluster into another newly created cluster.
* *[Recovering from a single disk failure using JBOD|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html]*
Steps for recovering from a single disk failure in a disk array using JBOD (just a bunch of disks).
h1. About snapshots
DataStax Enterprise backs up data by taking a snapshot of all on-disk data files (SSTable files) stored in the data directory. You can take a snapshot of all keyspaces, a single keyspace, or a single table while the system is online.
Using a parallel ssh tool (such as pssh), you can snapshot an entire cluster. This provides an eventually consistent backup. Although no one node is guaranteed to be consistent with its replica nodes at the time a snapshot is taken, a restored snapshot resumes consistency using built-in consistency mechanisms.
After a system-wide snapshot is performed, you can enable incremental backups on each node to backup data that has changed since the last snapshot. Each time a memtable is flushed to disk and an SSTable is created, a hard link is copied into a {color:#374c51}/backups{color} subdirectory of the data directory (provided JNA is enabled). Compacted SSTables do not create hard links in {color:#374c51}/backups{color} because these SSTables do not contain any data that has not already been linked.
h1. Taking a snapshot
Snapshots are taken per node using the [nodetool snapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsSnapShot.html] command. To take a global snapshot, run the {{nodetool snapshot}} command with a parallel ssh utility, such as pssh.
A snapshot first [flushes|https://docs.datastax.com/en/dse/5.1/dse-arch/datastax_enterprise/dbInternals/dbIntHowDataMaintain.html] all in-memory writes to disk, then makes a hard link of the SSTable files for each keyspace. You must have enough free disk space on the node to accommodate making snapshots of your data files. A single snapshot requires little disk space. However, snapshots can cause your disk usage to grow more quickly over time because a snapshot prevents old obsolete data files from being deleted. After the snapshot is complete, you can move the backup files to another location if needed, or you can leave them in place.
Note: Restoring from a snapshot requires the table schema.
h2. Procedure
Run the nodetool snapshot command, specifying the hostname, JMX port, and keyspace. For example:
{color:#990073}$ {color}nodetool snapshot -t cycling_2017-3-9 cycling
Tarball and Installer No-Services path: {{installation_location{color:#009926}/resources/{color}cassandra/bin}}
h2. Results
The name of the snapshot directory appears:{{{color:#000080}Requested{color} {color:#000080}creating{color} {color:#000080}snapshot{color}({color:#000080}s{color}) {color:#000080}for{color} [cycling] {color:#000080}with{color} {color:#000080}snapshot{color} {color:#000080}name{color} [2015.07.17] {color:#000080}Snapshot{color} {color:#000080}directory{color}: {color:#000080}cycling_2017-3-9{color}}}
The snapshot files are created in {color:#374c51}data/keyspace_name/table_name-UUID/snapshots/snapshot_name{color} directory.
ls -1 data{color:#009926}/cycling/{color}cyclist_name-9e516080f30811e689e40725f37c761d{color:#009926}/snapshots/{color}cycling_2017-3-9
For all installations, the default location of the {color:#374c51}data{color} directory is {color:#374c51}/var/lib/cassandra/data{color}.
The data files extension is {color:#374c51}.db{color} and the full CQL to create the table is in the {color:#374c51}schema.cql{color} file.{{manifest.json mc-1-big-CompressionInfo.db mc-1-big-Data.db mc-1-big-Digest.crc32 mc-1-big-Filter.db mc-1-big-Index.db mc-1-big-Statistics.db mc-1-big-Summary.db mc-1-big-TOC.txt schema.cql}}
h1. Deleting snapshot files
When taking a snapshot, previous snapshot files are not automatically deleted. You should remove old snapshots that are no longer needed.
The [nodetool clearsnapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsClearSnapShot.html] command removes all existing snapshot files from the snapshot directory of each keyspace. You should make it part of your back-up process to clear old snapshots before taking a new one.
h2. Procedure
# To delete all snapshots for a node, run the nodetool {color:#374c51}clearsnapshot{color} command. For example:
nodetool -h localhost -p 7199 clearsnapshot
Tarball and Installer No-Services path: {{installation_location{color:#009926}/resources/{color}cassandra/bin}}
To delete snapshots on all nodes at once, run the nodetool clearsnapshot command using a parallel ssh utility.
# To delete a single snapshot, run the {color:#374c51}clearsnapshot{color} command with the snapshot name:
nodetool clearsnapshot -t <snapshot_name>
The file name and path vary according to the type of snapshot. See [nodetools snapshot |https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsSnapShot.html] for details about snapshot names and paths.
h1. Enabling incremental backups
When incremental backups are enabled (disabled by default), DataStax Enterprise hard-links each memtable-flushed SSTable to a backups directory under the keyspace data directory. This allows storing backups offsite without transferring entire snapshots. Also, incremental backups combined with snapshots to provide a dependable, up-to-date backup mechanism. Compacted SSTables do not create hard links in {color:#374c51}/backups{color} because these SSTables do not contain any data that has not already been linked.A snapshot at a point in time, plus all incremental backups and commit logs since that time form a compete backup.
As with snapshots, DataStax Enterprise does not automatically clear incremental backup files. DataStax recommends setting up a process to clear incremental backup hard-links each time a new snapshot is created.
h2. Procedure
Edit the [cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupIncremental.html#opsBackupIncremental__cassandrayaml] configuration file on each node in the cluster and change the value of [incremental_backups|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__incremental_backups] to true.
h1. Restoring from a snapshot
Methods for restoring from a snapshot.
Restoring a keyspace from a snapshot requires all snapshot files for the table, and if using incremental backups, any incremental backup files created after the snapshot was taken. Streamed SSTables (from repair, decommission, and so on) are also hardlinked and included.
Note: Restoring from snapshots and incremental backups temporarily causes intensive CPU and I/O activity on the node being restored.
h2. Restoring from local nodes
This method copies the SSTables from the snapshots directory into the correct data directories.
# Make sure the table schema exists and is the same as when the snapshot was created.The nodetool snapshot command creates a table schema in the output directory. If the table does not exist, recreate it using the schema.cql file.
# If necessary, [{color:#0066cc}truncate{color}|https://docs.datastax.com/en/dse/5.1/cql/cql/cql_reference/cql_commands/cqlTruncate.html] the table.
Note: You may not need to truncate under certain conditions. For example, if a node lost a disk, you might restart before restoring so that the node continues to receive new writes before starting the restore procedure.
Truncating is usually necessary. For example, if there was an accidental deletion of data, the tombstone from that delete has a later write timestamp than the data in the snapshot. If you restore without truncating (removing the tombstone), the database continues to shadow the restored data. This behavior also occurs for other types of overwrites and causes the same problem.
# Locate the most recent snapshot folder. For example:/var/lib/cassandra/data/keyspace_name/table_name-UUID/snapshots/snapshot_name
# Copy the most recent snapshot SSTable directory to the /var/lib/cassandra/data/keyspace/table_name-UUID directory.
For all installations, the default location of the data directory is /var/lib/cassandra/data.
# Run [{color:#0066cc}nodetool refresh{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRefresh.html].
h2. Restoring from centralized backups
This method uses [{color:#0066cc}sstableloader{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsBulkloader.html] to restore snapshots.
# Verify that the SSTable version is compatible with the current version of DSE:
## Locate the version in the file names.
Use the version number in the SSTable file name to determine compatibility and upgrade requirements. The first two letters of the file name is the version, where the first letter indicates a major version and the second letter indicates a minor version. For example, the following SSTable version is mc:{{data/cycling/cyclist_expenses-2d955621194c11e7a38d9504a063a84e/mc-6-big-Data.db}}
## Using the correct DSE version of {{sstableupgrade}}, to create a compatible version:
|SSTable compatibility and upgrade version|
||DSE version||SSTable version||[{color:#0066cc}sstableloader{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsBulkloader.html] supported version||[{color:#0066cc}sstableupgrade{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/ToolsSSTableupgrade.html] and [{color:#0066cc}nodetool upgradesstables{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsUpgradeSstables.html] supported version||
|4.0|jb|jb only (must upgrade to load)|i* only|
|4.5|
|4.6|
|4.7|ka|ka and jb|j* only|
|4.8|
|5.0.x|ma|ka and ma|k* only|
|5.0.x|mb|ka, ma, and mb|ka and ma|
|5.0.x|mc|ka, ma, mb, and mc|ka, ma, and mb|
|5.1.x|
# Make sure the table schema exists and is the same as when the snapshot was created.The nodetool snapshot command creates a table schema in the output directory. If the table does not exist, recreate it using the schema.cql file.
# If necessary, [{color:#0066cc}truncate{color}|https://docs.datastax.com/en/dse/5.1/cql/cql/cql_reference/cql_commands/cqlTruncate.html] the table.
Note: You may not need to truncate under certain conditions. For example, if a node lost a disk, you might restart before restoring so that the node continues to receive new writes before starting the restore procedure.
Truncating is usually necessary. For example, if there was an accidental deletion of data, the tombstone from that delete has a later write timestamp than the data in the snapshot. If you restore without truncating (removing the tombstone), the database continues to shadow the restored data. This behavior also occurs for other types of overwrites and causes the same problem.
# Restore the most recent snapshot using the [{color:#0066cc}sstableloader{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsBulkloader.html] tool on the backed-up SSTables.
The sstableloader streams the SSTables to the correct nodes. You do not need to remove the commitlogs or drain or restart the nodes.
h1. Restoring a snapshot into a new cluster
Suppose you want to copy a snapshot of SSTable data files from a three node DataStax Enterprise cluster with vnodes enabled (128 tokens) and recover it on another newly created three node cluster (128 tokens). The token ranges will not match, because the token ranges cannot be exactly the same in the new cluster. You need to specify the tokens for the new cluster that were used in the old cluster.
Note: This procedure assumes you are familiar with [restoring a snapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupSnapshotRestore.html] and configuring and [initializing a cluster|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/production/initDSETOC.html].
h2. Procedure
To recover the snapshot on the new cluster:
# From the old cluster, retrieve the list of tokens associated with each node's IP:
{color:#990073}$ {color}nodetool ring | grep ip_address_of_node | awk {color:#4169e1}'\{print $NF ","}'{color} | xargs
# In the [cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml] file for each node in the new cluster, add the list of tokens you obtained in the previous step to the [initial_token|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__initial_token] parameter using the same {color:#374c51}num_tokens{color} setting as in the old cluster.
# Make any other necessary changes in the new cluster's [cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml] and property files so that the new nodes match the old cluster settings. Make sure the seed nodes are set for the new cluster.
# Clear the system table data from each new node:
{color:#990073}$ {color}{color:#0086b3}sudo{color} rm -rf /var/lib/cassandra/data/system/*
This allows the new nodes to use the initial tokens defined in the [cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml] when they restart.
# Start each node using the specified list of token ranges in new cluster's [cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml]:
{{initial_token: -9211270970129494930, -9138351317258731895, -8980763462514965928, ...}}
# Create schema in the new cluster. All the schemas from the old cluster must be reproduced in the new cluster.
# [Stop the node|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startStopDseToc.html]. Using {{nodetool refresh}} is unsafe because files within the data directory of a running node can be silently overwritten by identically named just-flushed SSTables from memtable flushes or compaction. Copying files into the data directory and restarting the node will not work for the same reason.
# [Restore the SSTable files snapshotted|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupSnapshotRestore.html] from the old cluster onto the new cluster using the same directories, w{color:#374c51}hile noting that the UUID component of target directory names has changed. Without restoration, the new cluster will not have data to read upon restart. {color}
# {color:#374c51}[Restart the node|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startStopDseToc.html].{color}
Recovering from a single disk failure using JBOD
Steps for recovering from a single disk failure in a disk array using JBOD (just a bunch of disks).
Steps for recovering from a single disk failure in a disk array using JBOD (just a bunch of disks).
DataStax Enterprise might not fail from the loss of one disk in a JBOD array, but some reads and writes may fail when:The operation's consistency level is ALL.
The data being requested or written is stored on the defective disk.
The data to be compacted is on the defective disk.
It's possible that you can simply replace the disk, restart DataStax Enterprise, and run nodetool repair. However, if the disk crash corrupted system table, you must remove the incomplete data from the other disks in the array. The procedure for doing this depends on whether the cluster uses vnodes or single-token architecture.
Procedure
If a disk fails on a node in a cluster using DataStax Enterprise 5.0 or earlier, replace the node.
Verify that the node has a defective disk and identify the disk, by checking the logs on the affected node.
Disk failures are logged in FILE NOT FOUND entries, which identifies the mount point or disk that has failed.
If the node is still running, stop DSE and shut down the node.
Replace the defective disk and restart the node.
If the node cannot restart:
Try restarting DataStax Enterprise without bootstrapping the node:
Package and Installer-Services installations: Add the following option to cassandra-env.sh [file:JVM_OPTS=]"$JVM_OPTS -Dcassandra.allow_unsafe_replace=true
Starting DataStax Enterprise as a service.
After the node bootstraps, remove the -Dcassandra.allow_unsafe_replace=true parameter from cassandra-env.sh.
Starting DataStax Enterprise as a service.
Tarball and Installer-No Services installations:
Start DataStax Enterprise with this option:$ sudo bin/dse cassandra Dcassandra.allow_unsafe_replace=true
Tarball and Installer No-Services path: installation_location
If DataStax Enterprise restarts, run nodetool repair on the node. If not, replace the node.
If the repair succeeds, the node is restored to production. Otherwise, go to 7 or 8.
For a cluster using vnodes: On the affected node, clear the system directory on each functioning drive.
Example for a node with a three disk JBOD array:$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
If mnt1 has failed:$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
Restart DataStax Enterprise without bootstrapping as described in 4:
$ -Dcassandra.allow_unsafe_replace=true
Run nodetool repair on the node.
If the repair succeeds, the node is restored to production. If not, replace the dead node.
For a cluster single-token nodes: On one of the cluster's working nodes, run nodetool ring to retrieve the list of the repaired node's tokens:
$ nodetool ring | grep ip_address_of_node | awk '
{print $NF ","}
' | xargs
Copy the output of the nodetool ring into a spreadsheet (space-delimited).
Edit the output, keeping the list of tokens and deleting the other columns.
On the node with the new disk, open the cassandra.yaml file and add the tokens (as a comma-separated list) to the initial_token property.
Change any other non-default settings in the new nodes to match the existing nodes. Use the diff command to find and merge any differences between the nodes.
If the repair succeeds, the node is restored to production. If not, replace the node.
On the affected node, clear the system directory on each functioning drive.
Example for a node with a three disk JBOD array:$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
If mnt1 has failed:$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
Restart DataStax Enterprise without bootstrapping as described in 4:
$ -Dcassandra.allow_unsafe_replace=true
Run nodetool repair on the node.
If the repair succeeds, the node is restored to production. If not, replace the node.
h1. Recovering from a single disk failure using JBOD
Steps for recovering from a single disk failure in a disk array using JBOD (just a bunch of disks).
Steps for recovering from a single disk failure in a disk array using JBOD (just a bunch of disks).
DataStax Enterprise might not fail from the loss of one disk in a JBOD array, but some reads and writes may fail when: * The operation's consistency level is ALL.
* The data being requested or written is stored on the defective disk.
* The data to be compacted is on the defective disk.
It's possible that you can simply replace the disk, restart DataStax Enterprise, and run nodetool repair. However, if the disk crash corrupted system table, you must remove the incomplete data from the other disks in the array. The procedure for doing this depends on whether the cluster uses [{color:#0066cc}vnodes{color}|https://docs.datastax.com/en/glossary/doc/glossary/gloss_vnode.html] or single-token architecture.
h2. Procedure
If a disk fails on a node in a cluster using DataStax Enterprise 5.0 or earlier, [{color:#0066cc}replace the node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
# Verify that the node has a defective disk and identify the disk, by checking the [{color:#0066cc}logs{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configLoggingLevels.html] on the affected node.
Disk failures are logged in {{FILE NOT FOUND}} entries, which identifies the mount point or disk that has failed.
# If the node is still running, [{color:#0066cc}stop DSE{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startStopDseToc.html] and shut down the node.
# Replace the defective disk and restart the node.
# If the node cannot restart:
## Try restarting DataStax Enterprise without bootstrapping the node:
Package and Installer-Services installations: # Add the following option to [{color:#0066cc}cassandra-env.sh{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__cassandraenvsh] file:{{JVM_OPTS="$JVM_OPTS -Dcassandra.allow_unsafe_replace=true}}
# [{color:#0066cc}Starting DataStax Enterprise as a service{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startDseService.html].
# After the node bootstraps, remove the {{-Dcassandra.allow_unsafe_replace=true}} parameter from cassandra-env.sh.
# [{color:#0066cc}Starting DataStax Enterprise as a service{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startDseService.html].
Tarball and Installer-No Services installations:
*** Start DataStax Enterprise with this option:
$ sudo bin/dse cassandra Dcassandra.allow_unsafe_replace=trueTarball and Installer No-Services path: {{installation_location}}
# If DataStax Enterprise restarts, run [{color:#0066cc}nodetool repair{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html] on the node. If not, [{color:#0066cc}replace the node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
# If the repair succeeds, the node is restored to production. Otherwise, go to [{color:#0066cc}7{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__vnode-used] or [{color:#0066cc}8{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__single-token-nodes-used].
# For a cluster using vnodes:
## On the affected node, clear the system directory on each functioning drive.
Example for a node with a three disk JBOD array:$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
If {{mnt1}} has failed:
$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
## Restart DataStax Enterprise without bootstrapping as described in [{color:#0066cc}4{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__restart-w-out-boots]:
$ -Dcassandra.allow_unsafe_replace=true
## Run [{color:#0066cc}nodetool repair{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html] on the node.
If the repair succeeds, the node is restored to production. If not, [{color:#0066cc}replace the dead node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
# For a cluster single-token nodes:
## On one of the cluster's working nodes, run [{color:#0066cc}nodetool ring{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRing.html] to retrieve the list of the repaired node's tokens:
$ nodetool ring | grep ip_address_of_node | awk ' \{print $NF ","}' | xargs
## Copy the output of the nodetool ring into a spreadsheet (space-delimited).
## Edit the output, keeping the list of tokens and deleting the other columns.
## On the node with the new disk, open the [{color:#0066cc}cassandra.yaml{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__cassandrayaml] file and add the tokens (as a comma-separated list) to the [{color:#0066cc}initial_token{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__initial_token] property.
## Change any other non-default settings in the new nodes to match the existing nodes. Use the diff command to find and merge any differences between the nodes.
If the repair succeeds, the node is restored to production. If not, [{color:#0066cc}replace the node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
## On the affected node, clear the system directory on each functioning drive.
Example for a node with a three disk JBOD array:$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
If {{mnt1}} has failed:
$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
## Restart DataStax Enterprise without bootstrapping as described in [{color:#0066cc}4{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__restart-w-out-boots]:
$ -Dcassandra.allow_unsafe_replace=true
## Run [{color:#0066cc}nodetool repair{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html] on the node.
If the repair succeeds, the node is restored to production. If not, [{color:#0066cc}replace the node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
cassandra.yaml
The location of the cassandra.yaml file depends on the type of installation:
|Package installations
Installer-Services installations|/etc/dse/cassandra/cassandra.yaml|
|Tarball installations
Installer-No Services installations|installation_location/resources/cassandra/conf/cassandra.yaml|
cassandra-rackdc.properties
The location of the cassandra-rackdc.properties file depends on the type of installation:
|Package installations
Installer-Services installations|/etc/dse/cassandra/cassandra-rackdc.properties|
|Tarball installations
Installer-No Services installations|installation_location/resources/cassandra/conf/cassandra-rackdc.properties|
cassandra-topology.properties
The location of the cassandra-topology.properties file depends on the type of installation:
|Package installations
Installer-Services installations|/etc/dse/cassandra/cassandra-topology.properties|
|Tarball installations
Installer-No Services installations|installation_location/resources/cassandra/conf/cassandra-topology.properties|
cassandra-env.sh
The location of the cassandra-env.sh file depends on the type of installation:
|Package installations
Installer-Services installations|/etc/dse/cassandra/cassandra-env.sh|
|Tarball installations
Installer-No Services installations|installation_location/resources/cassandra/conf/cassandra-env.sh|
*
> The Backups web page on the web site is empty
> ---------------------------------------------
>
> Key: CASSANDRA-14272
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14272
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation and Website
> Reporter: Kenneth Brotman
> Priority: Major
>
> [http://cassandra.apache.org/doc/latest/operating/backups.html]
> is empty. Please contribute content. Myself or someone else will take it from there.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org