[jira] [Commented] (KUDU-3731) Long-running smoke test

["Gabriella Lotz (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18074284#comment-18074284 ] 

Gabriella Lotz commented on KUDU-3731:
--------------------------------------

h2. Two TServers Stopped Simultaneously

Setup: Stopped tserver-2 and tserver-3 back-to-back. Observed graceful degradation and recovery.
h3. Findings:

While going down:

When tserver-2 went UNAVAILABLE first, Raft elected new leaders on the remaining servers. ksck showed tserver-3 gaining 3 leaders (11→14) as tserver-2's tablets re-elected. The leader rebalancer fired at this moment and successfully transferred 3 leaders for user_events to re-balance across the 2 survivors.

Once both tservers were UNAVAILABLE, the replica rebalancer error message changed, from the generic IllegalState: not all tservers available to a UUID-specific NotFound: tserver 4e9eadc3... not available for placement. The rebalancer still halted correctly, the error message difference is an observable but harmless implementation detail.

Leader rebalancer with 2 tservers down: Ran its normal 60-second cycle, iterated all tables, computed zero valid leader transfers (no non-presumed-dead destinations available), and finished cleanly with leader transfer count: 0.

Replica rebalancer with 2 tservers down: Consistently returned NotFound errors every interval and made no moves. Correct defensive behavior, no moves during loss of quorum risk.

Recovery (tservers restarted at 15:07):

The master logged cstate changes per tablet as Raft groups re-elected leaders:
T 7704fd89... term changed from 2 to 3, leader changed from tserver-1 to tserver-2
Leader distribution immediately after restart: 12/8/12.
Within ~2 minutes, the leader rebalancer corrected to 11/10/11 - fully balanced.
Replica rebalancer NotFound errors ceased once both tservers were registered and available for placement.
Summary: The auto rebalancer handles total loss of quorum gracefully.
h2. Maintenance Mode & Quiescing
h3. Maintenance Mode

Tested using kudu tserver state enter_maintenance.

The auto replica rebalancer correctly blocked all moves while the server was in maintenance mode. The master log showed "Could not retrieve cluster info: not all tservers available for tablet placement" every 60 seconds (matching --auto_rebalancing_interval_seconds=60), confirming the rebalancer skipped every round. The cluster remained fully HEALTHY throughout.
h3. Quiescing

Status: Partially tested. The runs below were affected by repeated quiesce/stop cycles on the same session and an idle cluster with no active write traffic. A clean end-to-end run (fresh cluster state, loadgen running during quiesce) is needed.
h4. What Consistently Worked

Flag propagation: quiesce start takes effect immediately. ksck remote_server_quiescing Raft guard: The quiescing tserver advertises its state in Raft heartbeat responses. The leader correctly refuses to transfer leadership to a quiescing server, so no leaders accidentally land on it via Raft's own mechanism.
quiesce stop recovery: Un-quiescing restores eligibility immediately. The leader rebalancer re-balanced leadership within 1–2 cycles (~60–120 s).
h4. Observation 1

Leader Rebalancer Tries to Push Leaders to Quiescing Server (April 13)
On the April 13 run, after the quiescing tserver drained to 0 leaders, the leader rebalancer treated it as the most under-loaded server and fired LeaderStepDown RPCs every 60 seconds to transfer leaders back to it. These transfers failed silently. The rebalancer wasted 8 RPCs per cycle for as long as quiescing was active.

The likely cause is that RunLeaderRebalanceForTable builds exclude_dest_uuids only from MAINTENANCE_MODE tservers and has no equivalent check for quiescing state.

Needs confirmation: Reproduce with a clean quiesce run, watch for leader transfer count > 0 cycles after the drain completes, and confirm the transfers are targeting the quiescing server.
h4. Observation 2

Leader Drain Did Not Complete on Idle Cluster (Unclear Root Cause, April 17)
Across three separate quiesce attempts on April 17, leaders on tserver-1 did not drain. On April 13, when loadgen write traffic was active, the drain completed for all tablets. On April 17, fully idle cluster, several prior quiesce/stop cycles, it silently stalled.

What makes this uncertain:

Only one truly clean run exists (April 13, with loadgen running).
The April 17 runs were contaminated by repeated quiesce/stop cycles, a failed loadgen write attempt (20000 schema-mismatch errors), and multiple leader rebalancer cycles between attempts.
h4. Plan for clean test:

Fresh cluster state: no prior quiesce/stop cycles.
Start a light background write workload before quiescing.
Quiesce the tserver. Confirm drain completes (all leaders → 0).
Stop the workload. Quiesce again on the now-idle cluster.

> Long-running smoke test
> -----------------------
>
>                 Key: KUDU-3731
>                 URL: https://issues.apache.org/jira/browse/KUDU-3731
>             Project: Kudu
>          Issue Type: Sub-task
>            Reporter: Gabriella Lotz
>            Assignee: Gabriella Lotz
>            Priority: Major
>
> h4. Step 0: Start a cluster with 1 master and 3 tablet servers.
> h4. Step 1: Set the following flags.
> --auto_rebalancing_enabled=true
> --auto_rebalancing_interval_seconds=60
> --auto_leader_rebalancing_enabled=true
> h4. Step 2: Create the following tables.
> {code:java}
> kudu table create <master> '{
>   "table_name": "user_events",
>   "schema": {
>     "columns": [
>       {"column_name": "user_id",  "column_type": "STRING", "is_nullable": false},
>       {"column_name": "event_id", "column_type": "INT64",  "is_nullable": false},
>       {"column_name": "data",     "column_type": "STRING", "is_nullable": true}
>     ],
>     "key_column_names": ["user_id", "event_id"]
>   },
>   "partition": {
>     "hash_partitions": [{"columns": ["user_id", "event_id"], "num_buckets": 8}]
>   },
>   "num_replicas": 3
> }'
> kudu table create <master> '{
>   "table_name": "time_series_table",
>   "schema": {
>     "columns": [
>       {"column_name": "ts",        "column_type": "UNIXTIME_MICROS", "is_nullable": false},
>       {"column_name": "sensor_id", "column_type": "STRING",          "is_nullable": false},
>       {"column_name": "value",     "column_type": "DOUBLE",          "is_nullable": true}
>     ],
>     "key_column_names": ["ts", "sensor_id"]
>   },
>   "partition": {
>     "range_partition": {
>       "columns": ["ts"],
>       "range_bounds": [
>         {"upper_bound": {"bound_type": "exclusive", "bound_values": ["1704067200000000"]}},
>         {
>           "lower_bound": {"bound_type": "inclusive", "bound_values": ["1704067200000000"]},
>           "upper_bound": {"bound_type": "exclusive", "bound_values": ["1735689600000000"]}
>         },
>         {"lower_bound": {"bound_type": "inclusive", "bound_values": ["1735689600000000"]}}
>       ]
>     }
>   },
>   "num_replicas": 3
> }' {code}
> h4. Step 3: Start loadgen in tmux
> tmux new -s smoke
> Inside tmux:
> while true; do
>   kudu perf loadgen <master> \
>     --table_name=user_events \
>     --num_threads=4 \
>     --num_rows_per_thread=500000 \
>     --flush_per_n_rows=1000 \
>     --run_cleanup # cleanup so that disk doesn't fill up
>   sleep 10
> done
> h4. Step 4: Monitor findings for 2 weeks. (start 2026.03.13.)
>  # Is the cluster healthy?
> kudu cluster ksck <master>
>  # Check whether auto-rebalancer is running.
> grep -i rebalanc /var/log/kudu/kudu-master.INFO | tail -20



--
This message was sent by Atlassian Jira
(v8.20.10#820010)