You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Capwell (Jira)" <ji...@apache.org> on 2022/03/29 23:52:00 UTC
[jira] [Comment Edited] (CASSANDRA-15399) Add ability to track state in repair
[ https://issues.apache.org/jira/browse/CASSANDRA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514369#comment-17514369 ]
David Capwell edited comment on CASSANDRA-15399 at 3/29/22, 11:51 PM:
----------------------------------------------------------------------
There are 2 types of tables: *coordinator*, *participate*; here are examples from both sides
*Participate* (host involved in the repair, this may be the coordinator, but would include all peers involved in the repair)
{code}
cqlsh> SELECT * FROM system_views.repair_participates WHERE tables CONTAINS 'tlp_stress.clusteringslicing' ALLOW FILTERING;
@ Row 1
-------------------------+----------------------------------------
id | 584eca30-afb6-11ec-9bff-cd38a4d988d3
duration_millis | 14773
failure_cause | null
global | True
incremental | False
last_updated_at | 2022-03-29 23:17:18.993000+0000
preview_kind | NONE
ranges | ['(-3074457345618258603,0]']
repaired_at | null
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:17:04.220000+0000
state_skipped_timestamp | null
state_success_timestamp | 2022-03-29 23:17:18.993000+0000
success_message | Cleanup message recieved
tables | {'tlp_stress.clusteringslicing'}
validations | {0034e064-b54a-384f-99d0-af1bf42844a4}
(1 rows)
cqlsh> SELECT * FROM system_views.repair_validations WJERE id=0034e064-b54a-384f-99d0-af1bf42844a4; -- id from validations field above
@ Row 1
-------------------------------+--------------------------------------
id | 0034e064-b54a-384f-99d0-af1bf42844a4
duration_millis | 14688
estimated_partitions | 897
estimated_total_bytes | 2321697131
failure_cause | null
initiator | /127.0.0.1:7000
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:17:18.916000+0000
partitions_processed | 181
progress_percentage | 100
ranges | ['(-3074457345618258603,0]']
repair_id | 584eca30-afb6-11ec-9bff-cd38a4d988d3
session_id | 58509ef0-afb6-11ec-9bff-cd38a4d988d3
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:17:04.228000+0000
state_sending_trees_timestamp | 2022-03-29 23:17:18.916000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:17:04.299000+0000
state_success_timestamp | 2022-03-29 23:17:18.916000+0000
status | success
success_message | null
table_name | clusteringslicing
(1 rows)
{code}
What is lacking? Linking with system_views.streaming, my plan is to update repair/streaming to support deep linking after this patch is merged (need to modify network messages for remote repair + streaming, so repair knows stream ids, and stream knows repair id)
*Coordinator*
{code}
cqlsh> SELECT * FROM system_views.repairs WHERE table_names CONTAINS 'clusteringslicing' ALLOW FILTERING;
@ Row 1
---------------------------------------+--------------------------------------------------
id | 34424630-afb6-11ec-ab62-fb846f906c0a
command_id | 2
duration_millis | 7799
failure_cause | null
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.530000+0000
options_column_families | {}
options_data_centers | {}
options_force_repair | False
options_hosts | {}
options_ignore_unreplicated_keyspaces | False
options_incremental | null
options_job_threads | 1
options_optimise_streams | False
options_parallelism | PARALLEL
options_preview_kind | NONE
options_primary_range | True
options_pull_repair | False
options_ranges | {'(6148914691236517205,-9223372036854775808]'}
options_subrange_repair | False
options_trace | False
participants | ['/127.0.0.1:7000', '/127.0.0.2:7000']
ranges | [['(6148914691236517205,-9223372036854775808]']]
sessions | {34441af0-afb6-11ec-ab62-fb846f906c0a}
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.731000+0000
state_prepare_complete_timestamp | 2022-03-29 23:16:03.741000+0000
state_prepare_start_timestamp | 2022-03-29 23:16:03.737000+0000
state_repair_complete_timestamp | 2022-03-29 23:16:11.529000+0000
state_repair_start_timestamp | 2022-03-29 23:16:03.742000+0000
state_setup_timestamp | 2022-03-29 23:16:03.731000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.732000+0000
state_success_timestamp | 2022-03-29 23:16:11.530000+0000
status | success
success_message | Repair completed successfully
table_names | ['clusteringslicing']
type | full
unfiltered_ranges | [['(6148914691236517205,-9223372036854775808]']]
(1 rows)
cqlsh> SELECT * FROM system_views.repair_sessions WHERE id=34441af0-afb6-11ec-ab62-fb846f906c0a; -- from sessions column above
@ Row 1
----------------------------+------------------------------------------------
id | 34441af0-afb6-11ec-ab62-fb846f906c0a
duration_millis | 7785
failure_cause | null
jobs | {f08d1aa3-04d5-3439-93a4-7beaf6bcd393}
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.528000+0000
ranges | ['(6148914691236517205,-9223372036854775808]']
repair_id | 34424630-afb6-11ec-ab62-fb846f906c0a
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.743000+0000
state_jobs_start_timestamp | 2022-03-29 23:16:03.746000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.743000+0000
state_success_timestamp | 2022-03-29 23:16:11.528000+0000
status | success
success_message | null
table_names | ['clusteringslicing']
(1 rows)
cqlsh> SELECT * FROM system_views.repair_jobs WHERE id=f08d1aa3-04d5-3439-93a4-7beaf6bcd393; -- from jobs column above
@ Row 1
-------------------------------------+------------------------------------------------
id | f08d1aa3-04d5-3439-93a4-7beaf6bcd393
duration_millis | 7779
failure_cause | null
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.526000+0000
ranges | ['(6148914691236517205,-9223372036854775808]']
repair_id | 34424630-afb6-11ec-ab62-fb846f906c0a
session_id | 34441af0-afb6-11ec-ab62-fb846f906c0a
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.747000+0000
state_skipped_timestamp | null
state_snapshot_complete_timestamp | null
state_snapshot_start_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.747000+0000
state_stream_start_timestamp | 2022-03-29 23:16:11.526000+0000
state_success_timestamp | 2022-03-29 23:16:11.526000+0000
state_validation_complete_timestamp | 2022-03-29 23:16:11.526000+0000
state_validation_start_timestamp | 2022-03-29 23:16:03.747000+0000
status | success
success_message | null
table_name | clusteringslicing
(1 rows)
{code}
Missing
* links to streaming
* repair_jobs/sessions doesn't show the participants, so requires a join with repairs
was (Author: dcapwell):
There are 2 types of tables: coordinator, participate; here are examples from both sides
*Participate* (host involved in the repair, this may be the coordinator, but would include all peers involved in the repair)
{code}
cqlsh> SELECT * FROM system_views.repair_participates WHERE tables CONTAINS 'tlp_stress.clusteringslicing' ALLOW FILTERING;
@ Row 1
-------------------------+----------------------------------------
id | 584eca30-afb6-11ec-9bff-cd38a4d988d3
duration_millis | 14773
failure_cause | null
global | True
incremental | False
last_updated_at | 2022-03-29 23:17:18.993000+0000
preview_kind | NONE
ranges | ['(-3074457345618258603,0]']
repaired_at | null
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:17:04.220000+0000
state_skipped_timestamp | null
state_success_timestamp | 2022-03-29 23:17:18.993000+0000
success_message | Cleanup message recieved
tables | {'tlp_stress.clusteringslicing'}
validations | {0034e064-b54a-384f-99d0-af1bf42844a4}
(1 rows)
cqlsh> SELECT * FROM system_views.repair_validations WJERE id=0034e064-b54a-384f-99d0-af1bf42844a4; -- id from validations field above
@ Row 1
-------------------------------+--------------------------------------
id | 0034e064-b54a-384f-99d0-af1bf42844a4
duration_millis | 14688
estimated_partitions | 897
estimated_total_bytes | 2321697131
failure_cause | null
initiator | /127.0.0.1:7000
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:17:18.916000+0000
partitions_processed | 181
progress_percentage | 100
ranges | ['(-3074457345618258603,0]']
repair_id | 584eca30-afb6-11ec-9bff-cd38a4d988d3
session_id | 58509ef0-afb6-11ec-9bff-cd38a4d988d3
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:17:04.228000+0000
state_sending_trees_timestamp | 2022-03-29 23:17:18.916000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:17:04.299000+0000
state_success_timestamp | 2022-03-29 23:17:18.916000+0000
status | success
success_message | null
table_name | clusteringslicing
(1 rows)
{code}
What is lacking? Linking with system_views.streaming, my plan is to update repair/streaming to support deep linking after this patch is merged (need to modify network messages for remote repair + streaming, so repair knows stream ids, and stream knows repair id)
*Coordinator*
{code}
cqlsh> SELECT * FROM system_views.repairs WHERE table_names CONTAINS 'clusteringslicing' ALLOW FILTERING;
@ Row 1
---------------------------------------+--------------------------------------------------
id | 34424630-afb6-11ec-ab62-fb846f906c0a
command_id | 2
duration_millis | 7799
failure_cause | null
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.530000+0000
options_column_families | {}
options_data_centers | {}
options_force_repair | False
options_hosts | {}
options_ignore_unreplicated_keyspaces | False
options_incremental | null
options_job_threads | 1
options_optimise_streams | False
options_parallelism | PARALLEL
options_preview_kind | NONE
options_primary_range | True
options_pull_repair | False
options_ranges | {'(6148914691236517205,-9223372036854775808]'}
options_subrange_repair | False
options_trace | False
participants | ['/127.0.0.1:7000', '/127.0.0.2:7000']
ranges | [['(6148914691236517205,-9223372036854775808]']]
sessions | {34441af0-afb6-11ec-ab62-fb846f906c0a}
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.731000+0000
state_prepare_complete_timestamp | 2022-03-29 23:16:03.741000+0000
state_prepare_start_timestamp | 2022-03-29 23:16:03.737000+0000
state_repair_complete_timestamp | 2022-03-29 23:16:11.529000+0000
state_repair_start_timestamp | 2022-03-29 23:16:03.742000+0000
state_setup_timestamp | 2022-03-29 23:16:03.731000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.732000+0000
state_success_timestamp | 2022-03-29 23:16:11.530000+0000
status | success
success_message | Repair completed successfully
table_names | ['clusteringslicing']
type | full
unfiltered_ranges | [['(6148914691236517205,-9223372036854775808]']]
(1 rows)
cqlsh> SELECT * FROM system_views.repair_sessions WHERE id=34441af0-afb6-11ec-ab62-fb846f906c0a; -- from sessions column above
@ Row 1
----------------------------+------------------------------------------------
id | 34441af0-afb6-11ec-ab62-fb846f906c0a
duration_millis | 7785
failure_cause | null
jobs | {f08d1aa3-04d5-3439-93a4-7beaf6bcd393}
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.528000+0000
ranges | ['(6148914691236517205,-9223372036854775808]']
repair_id | 34424630-afb6-11ec-ab62-fb846f906c0a
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.743000+0000
state_jobs_start_timestamp | 2022-03-29 23:16:03.746000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.743000+0000
state_success_timestamp | 2022-03-29 23:16:11.528000+0000
status | success
success_message | null
table_names | ['clusteringslicing']
(1 rows)
cqlsh> SELECT * FROM system_views.repair_jobs WHERE id=f08d1aa3-04d5-3439-93a4-7beaf6bcd393; -- from jobs column above
@ Row 1
-------------------------------------+------------------------------------------------
id | f08d1aa3-04d5-3439-93a4-7beaf6bcd393
duration_millis | 7779
failure_cause | null
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.526000+0000
ranges | ['(6148914691236517205,-9223372036854775808]']
repair_id | 34424630-afb6-11ec-ab62-fb846f906c0a
session_id | 34441af0-afb6-11ec-ab62-fb846f906c0a
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.747000+0000
state_skipped_timestamp | null
state_snapshot_complete_timestamp | null
state_snapshot_start_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.747000+0000
state_stream_start_timestamp | 2022-03-29 23:16:11.526000+0000
state_success_timestamp | 2022-03-29 23:16:11.526000+0000
state_validation_complete_timestamp | 2022-03-29 23:16:11.526000+0000
state_validation_start_timestamp | 2022-03-29 23:16:03.747000+0000
status | success
success_message | null
table_name | clusteringslicing
(1 rows)
{code}
Missing
* links to streaming
* repair_jobs/sessions doesn't show the participants, so requires a join with repairs
> Add ability to track state in repair
> ------------------------------------
>
> Key: CASSANDRA-15399
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15399
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Repair
> Reporter: David Capwell
> Assignee: David Capwell
> Priority: Normal
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> To enhance the visibility in repair, we should expose internal state via virtual tables; the state should include coordinator as well as participant state (validation, sync, etc.)
> I propose the following tables:
> repairs - high level summary of the global state of repair; this should be called on the coordinator.
> {code:sql}
> CREATE TABLE repairs (
> id uuid,
> keyspace_name text,
> table_names frozen<list<text>>,
> ranges frozen<list<text>>,
> coordinator text,
> participants frozen<list<text>>,
> state text,
> progress_percentage float,
> last_updated_at_millis bigint,
> duration_micro bigint,
> failure_cause text,
> PRIMARY KEY ( (id) )
> )
> {code}
> repair_tasks - represents RepairJob and participants state. This will show if validations are running on participants and the progress they are making; this should be called on the coordinator.
> {code:sql}
> CREATE TABLE repair_tasks (
> id uuid,
> session_id uuid,
> keyspace_name text,
> table_name text,
> ranges frozen<list<text>>,
> coordinator text,
> participant text,
> state text,
> state_description text,
> progress_percentage float, -- between 0.0 and 100.0
> last_updated_at_millis bigint,
> duration_micro bigint,
> failure_cause text,
> PRIMARY KEY ( (id), session_id, table_name, participant )
> )
> {code}
> repair_validations - shows the state of the validation task and updated periodically while validation is running; this should be called on the participants.
> {code:sql}
> CREATE TABLE repair_validations (
> id uuid,
> session_id uuid,
> ranges frozen<list<text>>,
> keyspace_name text,
> table_name text,
> initiator text,
> state text,
> progress_percentage float,
> queue_duration_ms bigint,
> runtime_duration_ms bigint,
> total_duration_ms bigint,
> estimated_partitions bigint,
> partitions_processed bigint,
> estimated_total_bytes bigint,
> failure_cause text,
> PRIMARY KEY ( (id), session_id, table_name )
> )
> {code}
> The main reason for exposing virtual tables rather than exposing through durable tables is to make sure what is exposed is accurate. In cases of write failures or node failures, the durable tables could become in-accurate and could add edge cases where the repair is not running but the tables say it is; by relying on repair's internal in-memory bookkeeping, these problems go away.
> This jira does not try to solve the following:
> 1) repair resiliency - there are edge cases where repair hits an error and runs forever (at least from nodetool's perspective).
> 2) repair stream tracking - I have not learned the streaming side yet and what I see is multiple implementations exist, so seems like high scope. My hope is to punt from this jira and tackle separately.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org