You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Capwell (Jira)" <ji...@apache.org> on 2022/01/21 20:31:00 UTC
[jira] [Commented] (CASSANDRA-15399) Add ability to track state in repair
[ https://issues.apache.org/jira/browse/CASSANDRA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480257#comment-17480257 ]
David Capwell commented on CASSANDRA-15399:
-------------------------------------------
finally got time to come back to this, sent out a work-in-progress PR that shows coordinator state as well as validation; missing streaming...
here is sample output querying after running a repair
{code}
node2: SELECT * FROM system_views.repairs
id: 967551d0-7af5-11ec-abdf-517b0d807e7c
duration_millis: 555
failure_cause: null
keyspace_name: distributed_test_keyspace
last_updated_at: Fri Jan 21 12:06:15 PST 2022
participants: [/127.0.0.1:7012]
progress_percentage: 100.0
ranges: [[(-1,9223372036854775805], (9223372036854775805,-1]]]
state: success
state_init_timestamp: Fri Jan 21 12:06:15 PST 2022
state_prepare_complete_timestamp: Fri Jan 21 12:06:15 PST 2022
state_prepare_submit_timestamp: Fri Jan 21 12:06:15 PST 2022
state_repair_complete_timestamp: Fri Jan 21 12:06:15 PST 2022
state_repair_submit_timestamp: Fri Jan 21 12:06:15 PST 2022
state_setup_timestamp: Fri Jan 21 12:06:15 PST 2022
state_started_timestamp: Fri Jan 21 12:06:15 PST 2022
state_success_timestamp: Fri Jan 21 12:06:15 PST 2022
table_names: [simple_preview_sequential_true]
type: preview
unfiltered_ranges: [[(-1,9223372036854775805], (9223372036854775805,-1]]]
node2: SELECT * FROM system_views.repair_sessions
id: 968111a0-7af5-11ec-abdf-517b0d807e7c
duration_millis: 470
failure_cause: null
keyspace_name: distributed_test_keyspace
last_updated_at: Fri Jan 21 12:06:15 PST 2022
progress_percentage: 100.0
ranges: [(-1,9223372036854775805], (9223372036854775805,-1]]
repair_id: 967551d0-7af5-11ec-abdf-517b0d807e7c
state: success
state_failure_timestamp: null
state_init_timestamp: Fri Jan 21 12:06:15 PST 2022
state_jobs_submit_timestamp: Fri Jan 21 12:06:15 PST 2022
state_skipped_timestamp: null
state_started_timestamp: Fri Jan 21 12:06:15 PST 2022
state_success_timestamp: Fri Jan 21 12:06:15 PST 2022
table_names: [simple_preview_sequential_true]
node2: SELECT * FROM system_views.repair_jobs
id: 2e8c3999-04b0-3013-a351-f8a43283c7c2
duration_millis: 457
failure_cause: null
keyspace_name: distributed_test_keyspace
last_updated_at: Fri Jan 21 12:06:15 PST 2022
progress_percentage: 100.0
ranges: [(-1,9223372036854775805], (9223372036854775805,-1]]
repair_id: 967551d0-7af5-11ec-abdf-517b0d807e7c
session_id: 968111a0-7af5-11ec-abdf-517b0d807e7c
state: success
state_failure_timestamp: null
state_init_timestamp: Fri Jan 21 12:06:15 PST 2022
state_snapshot_complete_timestamp: Fri Jan 21 12:06:15 PST 2022
state_snapshot_submit_timestamp: Fri Jan 21 12:06:15 PST 2022
state_started_timestamp: Fri Jan 21 12:06:15 PST 2022
state_stream_submit_timestamp: Fri Jan 21 12:06:15 PST 2022
state_success_timestamp: Fri Jan 21 12:06:15 PST 2022
state_validation_complete_timestamp: Fri Jan 21 12:06:15 PST 2022
state_validation_submit_timestamp: Fri Jan 21 12:06:15 PST 2022
table_name: simple_preview_sequential_true
node1: SELECT * FROM system_views.repair_validations
id: 2e8c3999-04b0-3013-a351-f8a43283c7c2
duration_millis: 83
estimated_partitions: 129
estimated_total_bytes: 28
failure_cause: null
initiator: /127.0.0.2:7012
keyspace_name: distributed_test_keyspace
last_updated_at: Fri Jan 21 12:06:15 PST 2022
partitions_processed: 1
progress_percentage: 100.0
ranges: [(-1,9223372036854775805], (9223372036854775805,-1]]
repair_id: 967551d0-7af5-11ec-abdf-517b0d807e7c
session_id: 968111a0-7af5-11ec-abdf-517b0d807e7c
state: success
state_failure_timestamp: null
state_init_timestamp: Fri Jan 21 12:06:15 PST 2022
state_sending_trees_timestamp: Fri Jan 21 12:06:15 PST 2022
state_skipped_timestamp: null
state_started_timestamp: Fri Jan 21 12:06:15 PST 2022
state_success_timestamp: Fri Jan 21 12:06:15 PST 2022
table_name: simple_preview_sequential_true
{code}
> Add ability to track state in repair
> ------------------------------------
>
> Key: CASSANDRA-15399
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15399
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Repair
> Reporter: David Capwell
> Assignee: David Capwell
> Priority: Normal
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To enhance the visibility in repair, we should expose internal state via virtual tables; the state should include coordinator as well as participant state (validation, sync, etc.)
> I propose the following tables:
> repairs - high level summary of the global state of repair; this should be called on the coordinator.
> {code:sql}
> CREATE TABLE repairs (
> id uuid,
> keyspace_name text,
> table_names frozen<list<text>>,
> ranges frozen<list<text>>,
> coordinator text,
> participants frozen<list<text>>,
> state text,
> progress_percentage float,
> last_updated_at_millis bigint,
> duration_micro bigint,
> failure_cause text,
> PRIMARY KEY ( (id) )
> )
> {code}
> repair_tasks - represents RepairJob and participants state. This will show if validations are running on participants and the progress they are making; this should be called on the coordinator.
> {code:sql}
> CREATE TABLE repair_tasks (
> id uuid,
> session_id uuid,
> keyspace_name text,
> table_name text,
> ranges frozen<list<text>>,
> coordinator text,
> participant text,
> state text,
> state_description text,
> progress_percentage float, -- between 0.0 and 100.0
> last_updated_at_millis bigint,
> duration_micro bigint,
> failure_cause text,
> PRIMARY KEY ( (id), session_id, table_name, participant )
> )
> {code}
> repair_validations - shows the state of the validation task and updated periodically while validation is running; this should be called on the participants.
> {code:sql}
> CREATE TABLE repair_validations (
> id uuid,
> session_id uuid,
> ranges frozen<list<text>>,
> keyspace_name text,
> table_name text,
> initiator text,
> state text,
> progress_percentage float,
> queue_duration_ms bigint,
> runtime_duration_ms bigint,
> total_duration_ms bigint,
> estimated_partitions bigint,
> partitions_processed bigint,
> estimated_total_bytes bigint,
> failure_cause text,
> PRIMARY KEY ( (id), session_id, table_name )
> )
> {code}
> The main reason for exposing virtual tables rather than exposing through durable tables is to make sure what is exposed is accurate. In cases of write failures or node failures, the durable tables could become in-accurate and could add edge cases where the repair is not running but the tables say it is; by relying on repair's internal in-memory bookkeeping, these problems go away.
> This jira does not try to solve the following:
> 1) repair resiliency - there are edge cases where repair hits an error and runs forever (at least from nodetool's perspective).
> 2) repair stream tracking - I have not learned the streaming side yet and what I see is multiple implementations exist, so seems like high scope. My hope is to punt from this jira and tackle separately.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org