You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2026/05/30 16:04:39 UTC
(doris-skills) 05/06: test(verify): extend L1/L2 and verify on Doris 5.0.0
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/doris-skills.git
commit 14dd9cedb17accb3bfcbde288fa4230f42100263
Author: Mingyu Chen (Rayner) <yu...@selectdb.com>
AuthorDate: Sat May 30 21:39:16 2026 +0800
test(verify): extend L1/L2 and verify on Doris 5.0.0
L1 (now 29 cases, was 27) — re-verified green on 5.0.0/cloud:
- async-mv-refresh-interval-second.fail: EVERY 30 SECOND rejected (min 1 MINUTE)
- bool-default-null.fail: BOOLEAN DEFAULT "null" rejected (invalid boolean literal)
L2 (now 25 checks):
- add assertions for profile diff, profile history, and use (existence-only)
- add xassert (xfail/xpass) mechanism for known, version-specific doriscli gaps
- surfaced + tracked the query_stats.total_scan_rows parser gap, then confirmed
its fix (doris-cli dd3f417) via the xfail->xpass flip; reverted to a plain assert
Docs: README L1/L2 sections updated for 5.0.0; CLI-CONTRACT.md documents the
profile diff/history fields and notes use has no JSON contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
CLI-CONTRACT.md | 10 ++++
verify/README.md | 61 +++++++++++++++-------
verify/cli/run.sh | 56 ++++++++++++++++++--
.../async-mv-refresh-interval-second.fail.sql | 20 +++++++
verify/ddl/gotchas/bool-default-null.fail.sql | 13 +++++
5 files changed, 138 insertions(+), 22 deletions(-)
diff --git a/CLI-CONTRACT.md b/CLI-CONTRACT.md
index 58e84ab..d526cbb 100644
--- a/CLI-CONTRACT.md
+++ b/CLI-CONTRACT.md
@@ -38,6 +38,14 @@ present.
- `scanned_tables.<table>.{ddl, tablet_skew, total_rows}` (object keyed by table name)
- `served_by`, `fetch_attempts` (present on fetch failure)
+**`profile diff`** (shares `profile get`'s profile-fetch path)
+- `slow.{query_id, total_time_ms, operator_count}`, `fast.{…}`
+- `time_ratio`, `operator_diffs[]`
+
+**`profile history`** (p50/p99 trend from `audit_log`; independent of profile fetch)
+- `executions`, `entries[].{query_id, query_time_ms, state}`
+- `stats.query_time_ms.{p50, p99}` (present when ≥1 match)
+
**`tablet`**
- `model`, `bucket_key`, `bucket_count`, `sort_key`, `total_rows`
- `health.tablet_skew`
@@ -46,3 +54,5 @@ present.
**`auth status`**
- `mysql_status`, `http_status`, `http_probe`
- `backends[].alive` (empty / all-not-alive ⇒ compute suspended or unavailable)
+
+**`use`** — state-changing (switch the active env); no JSON-field contract.
diff --git a/verify/README.md b/verify/README.md
index 569af59..8c909e7 100644
--- a/verify/README.md
+++ b/verify/README.md
@@ -7,7 +7,7 @@ Apache Doris cluster and against `doriscli`. Three layers:
|---|---|---|---|
| **L1 — knowledge** | Every DDL template (T1–T5) and DDL gotcha in `doris-best-practices/SKILL.md` is accepted / rejected exactly as claimed | `mysql` client | ✅ `run.sh` |
| **L2 — CLI contract** | Every command + JSON field in `CLI-CONTRACT.md` really exists in `doriscli` | `doriscli --format json` + `jq` | ✅ `cli/run.sh` |
-| **L3 — behavior** | Triggering, evidence-first / safety guardrails, end-to-end DDL that loops back through L1 | `claude` + skill-creator eval harness | ⏳ planned |
+| **L3 — behavior** | Triggering, evidence-first / safety guardrails, end-to-end DDL that loops back through L1 | nested `claude -p` | 🟡 `behavior/run.sh` (v1) |
## Run L1
@@ -27,15 +27,16 @@ case fails, so it drops straight into CI.
## L1 coverage
-27 cases: the 5 DDL templates (T1–T5) and the `doris-best-practices/SKILL.md` §2
+29 cases: the 5 DDL templates (T1–T5) and the `doris-best-practices/SKILL.md` §2
"DDL hard constraints" matrix — each constraint as a reject case (often with a
-matching accept case): BOOLEAN-default quoting, `store_row_column` per model,
+matching accept case): BOOLEAN-default quoting (unquoted `TRUE` and the invalid
+`DEFAULT "null"` literal), `store_row_column` per model,
`compaction_policy=time_series` per model, inline-vs-PROPERTIES BloomFilter,
AGGREGATE column syntax (agg-fn order, `DEFAULT "null"` type rule), UNIQUE+RANGE
partition-column-in-key, key-columns-prefix order, AUTO PARTITION (`date_trunc`
+ empty parens), dynamic-partition needs a `PARTITION BY` clause,
`enable_unique_key_partial_update` as a (rejected) table property, and async-MV
-`REFRESH` syntax / `NOW()` nondeterministic-function rule.
+`REFRESH` syntax / minimum 1-MINUTE interval / `NOW()` nondeterministic-function rule.
Intentionally **not** L1 cases: claims that are advisory rather than DDL-rejections
(e.g. "don't set `dynamic_partition.buckets`" — Doris does not reject it), and the
@@ -46,21 +47,43 @@ runtime-diagnosis / sizing reference rules (those belong to L2/L3). Perf claims
`cli/run.sh` seeds a fact+dim dataset, then asserts every command + JSON field that
`../CLI-CONTRACT.md` hard-codes resolves against the live cluster (auto-detecting the
-doriscli binary and FE HTTP port). On Doris 4.1.1, **14 pass, 4 fail** — all four
-failures are `profile get` operator-level fields (`operators[]`, `query_stats.*`,
-`time_breakdown.plan`, `scanned_tables`). `auth status`, `tablet`, `sql`, and
-`profile list` fully satisfy the contract.
-
-Root cause of the `profile get` gap (doriscli 0.1.0, **not** the skills):
-1. Operator detail needs `profile_level=2` (default is `1`); doriscli's `--profile`
- only sets `enable_profile=true`, so the profile has no operator tree.
-2. With `profile_level=2` the operator-rich profile is large (~1.2 MB for a trivial
- join) and doriscli's `resp.json()` (`src/connection/http.rs:195`) fails to decode
- it — the body is valid JSON (verified with jq + python), so the bug is doriscli's.
-
-Effect: the skills' runtime-diagnosis layer (`cli-investigation.md`) reads those
-fields, so on 4.1.1 they come back null — exactly the "silently degrades to null"
-failure the contract warns about. The suite stays red here until doriscli is fixed.
+doriscli binary and FE HTTP port). It covers `auth status`, `tablet`, `sql --profile`,
+`profile list`, `profile get`, `profile diff`, `profile history`, and `use`
+(existence-only — it is state-changing, so it has no JSON contract and is not exercised
+in L2's stateless mode; the suite only asserts the command still exists).
+
+Latest run, **Doris 5.0.0 (cloud)** with doriscli post-`dd3f417`: **25 pass, 0 fail.**
+
+### How the suite earned its keep — the `total_scan_rows` gap (doriscli, **not** skills)
+
+On **4.1.1**, `profile get` returned null for `operators[]`, `time_breakdown.plan`,
+`scanned_tables`, and `query_stats.total_scan_rows` — one root cause: a `resp.json()`
+decode failure on the ~1.2 MB `profile_level=2` profile (`src/connection/http.rs:195`).
+On **5.0.0** the first three resolved, leaving `total_scan_rows` null via a *different*
+cause — the per-operator `RowsProduced` counter (which `total_scan_rows` is summed
+from, `summary.rs:284`) was dropped whenever a `PlanInfo` block preceded it in the
+merged profile. The suite marked it `xfail`; doris-cli fixed it in **`dd3f417`** ("keep
+per-operator counters when a PlanInfo block precedes them"); re-running here flipped the
+marker to `xpass`, confirming the fix, so it is back to a plain `assert`. The `xassert`
+(xfail / xpass) mechanism stays in `cli/run.sh` for the next version-specific gap.
+
+## L3 coverage (in progress)
+
+`behavior/run.sh` drives a nested `claude -p` (with `CLAUDECODE` unset) to check the
+skills *behave* as written, not just that the prose is correct. The skill text is
+injected via `--append-system-prompt` so the behavior under test is this repo's skill,
+isolated from any look-alike skill installed globally. The model is non-deterministic,
+so bump `SAMPLES` to repeat each case.
+
+Implemented (v1):
+- **Evidence-first hard gate** (`cli-investigation.md`): a slow-query prompt with no
+ evidence, run with `--disallowedTools Bash` so evidence can't be collected, must
+ yield an investigation plan + read-only commands and must **not** propose
+ DDL/MV/ALTER fixes. (`SHOW CREATE …` is excluded — it's a read-only evidence command.)
+
+Planned: connection-first rule, the `--profile` safety gate, sizing-as-total-vCPU
+(no per-node), brand-neutrality, triggering accuracy (needs a skill-isolated env to
+avoid the global look-alike), and the advisor→DDL→loop-back-through-L1 check.
## How a case works
diff --git a/verify/cli/run.sh b/verify/cli/run.sh
index 8de4848..9656e49 100755
--- a/verify/cli/run.sh
+++ b/verify/cli/run.sh
@@ -90,7 +90,7 @@ report="../reports/L2-$stamp.md"
echo "| result | command | contract field | note |"
echo "|---|---|---|---|"
} > "$report"
-pass=0; fail=0
+pass=0; fail=0; xfail=0; xpass=0
assert() { # assert <cmd-label> <field-label> <json> <jq-bool-expr> [note]
local cmd="$1" fld="$2" json="$3" expr="$4" note="${5:-}"
@@ -103,6 +103,22 @@ assert() { # assert <cmd-label> <field-label> <json> <jq-bool-expr> [note]
fi
}
+# xassert — like assert, but the field is a KNOWN doriscli limitation on this
+# cluster, so a miss is EXPECTED (xfail, non-fatal). If it unexpectedly resolves,
+# that is an xpass: the marker is stale and should be removed. Neither counts as
+# a hard failure, so the suite stays green on known-doriscli-only gaps but flags
+# loudly when reality changes.
+xassert() { # xassert <cmd> <field> <json> <jq-bool-expr> <reason>
+ local cmd="$1" fld="$2" json="$3" expr="$4" reason="$5"
+ if printf '%s' "$json" | jq -e "$expr" >/dev/null 2>&1; then
+ xpass=$((xpass+1)); warn "XPASS $cmd → $fld (now resolves — remove xfail?)"
+ echo "| 🎉 xpass | \`$cmd\` | $fld | now resolves on this cluster — remove the xfail marker ($reason) |" >> "$report"
+ else
+ xfail=$((xfail+1)); dim "xfail $cmd → $fld (known: $reason)"
+ echo "| 🟡 xfail | \`$cmd\` | $fld | known doriscli gap — $reason |" >> "$report"
+ fi
+}
+
# --- auth status -----------------------------------------------------------
J=$(dcli auth status)
assert "auth status" "mysql_status" "$J" '.mysql_status != null'
@@ -125,6 +141,10 @@ QJOIN='SELECT d.category,count(*) c,sum(s.amount) t FROM '"$SCRATCH_DB"'.sales s
J=$(dcli sql "$QJOIN" --profile --set 'profile_level=2')
assert "sql --profile" "query_id" "$J" '.query_id != null'
QID=$(printf '%s' "$J" | jq -r '.query_id // empty')
+# a second profiled query so `profile diff` has two query_ids to compare
+QSCAN='SELECT count(*) c FROM '"$SCRATCH_DB"'.sales WHERE amount>50'
+J2=$(dcli sql "$QSCAN" --profile --set 'profile_level=2')
+QID2=$(printf '%s' "$J2" | jq -r '.query_id // empty')
# --- profile list ----------------------------------------------------------
J=$(dcli profile list --limit 5)
@@ -135,15 +155,45 @@ assert "profile list" "[].query_id" "$J" '(type=="array") and (.[0].query_id !=
J=""; for _ in 1 2 3 4 5; do J=$(dcli profile get "$QID"); printf '%s' "$J" | jq -e '(.operators|length)>0' >/dev/null 2>&1 && break; done
assert "profile get" "summary.total_time_ms" "$J" '.summary.total_time_ms != null'
assert "profile get" "operators[]" "$J" '(.operators|type=="array") and ((.operators|length)>0)' "needs profile_level=2 AND doriscli must decode a large profile"
+# Fixed in doriscli dd3f417 (parser: keep per-operator counters when a PlanInfo
+# block precedes them) — was xfail on the pre-fix 0.1.x build, verified resolved
+# on 5.0.0 cloud. Back to a normal assert. (xassert helper retained for future gaps.)
assert "profile get" "query_stats.total_scan_rows" "$J" '.query_stats.total_scan_rows != null'
assert "profile get" "time_breakdown.plan" "$J" '.time_breakdown.plan != null'
assert "profile get" "scanned_tables.<t>" "$J" '(.scanned_tables|type=="object") and ((.scanned_tables|length)>0)'
+# --- profile diff (two profiled queries; shares profile get's fetch path) --
+J=""; for _ in 1 2 3 4 5; do J=$(dcli profile diff "$QID" "$QID2"); printf '%s' "$J" | jq -e '.slow.query_id != null' >/dev/null 2>&1 && break; done
+assert "profile diff" "slow.query_id" "$J" '.slow.query_id != null'
+assert "profile diff" "fast.query_id" "$J" '.fast.query_id != null'
+assert "profile diff" "operator_diffs[]" "$J" '(.operator_diffs|type=="array")'
+assert "profile diff" "time_ratio" "$J" '.time_ratio != null'
+
+# --- profile history (p50/p99 trend from audit_log; independent fetch path) -
+# Shape check: command + JSON contract resolve even with 0 matches
+# ({executions:0, entries:[]}). Fails only if audit_log is inaccessible.
+J=$(dcli profile history "sales" --days 7 --limit 5)
+assert "profile history" "executions" "$J" '.executions != null'
+assert "profile history" "entries[]" "$J" '(.entries|type=="array")'
+
+# --- use (env switch) — a state change, not a JSON-contract command; not
+# exercised in L2's stateless mode (HOST/USER set => no named env to switch).
+# Assert only that the command still exists in doriscli's CLI surface.
+if "$DCLI" use --help >/dev/null 2>&1; then
+ pass=$((pass+1)); ok "PASS use → command exists (stateless: switch not exercised)"
+ echo "| ✅ pass | \`use\` | command exists | stateless harness: env-switch not exercised |" >> "$report"
+else
+ fail=$((fail+1)); err "FAIL use → command missing (use --help non-zero)"
+ echo "| ❌ FAIL | \`use\` | command exists | \`use --help\` did not exit 0 |" >> "$report"
+fi
+
# --- cleanup ---------------------------------------------------------------
[ "${KEEP:-0}" = 1 ] || "${MYSQL[@]}" -e "DROP DATABASE IF EXISTS \`$SCRATCH_DB\`" >/dev/null 2>&1
-{ echo; echo "**$pass passed, $fail failed** — $((pass+fail)) contract checks."; } >> "$report"
+{ echo; echo "**$pass passed, $fail failed, $xfail xfail, $xpass xpass** — $((pass+fail+xfail+xpass)) contract checks."; } >> "$report"
log ""
-[ $fail -eq 0 ] && ok "L2: $pass passed, $fail failed." || err "L2: $pass passed, $fail failed."
+sumline="L2: $pass passed, $fail failed, $xfail xfail, $xpass xpass."
+[ $fail -eq 0 ] && ok "$sumline" || err "$sumline"
+[ "$xpass" -gt 0 ] && warn " ↳ $xpass xpass: a known-fail now resolves — review/remove the xfail marker."
log "report: ${report#../}"
[ $fail -eq 0 ]
diff --git a/verify/ddl/gotchas/async-mv-refresh-interval-second.fail.sql b/verify/ddl/gotchas/async-mv-refresh-interval-second.fail.sql
new file mode 100644
index 0000000..effa5e5
--- /dev/null
+++ b/verify/ddl/gotchas/async-mv-refresh-interval-second.fail.sql
@@ -0,0 +1,20 @@
+-- ref: doris-best-practices/SKILL.md:95
+-- desc: async MV refresh interval below the 1-MINUTE minimum (EVERY 30 SECOND) must be rejected
+-- min_version: 2.1
+-- mode: any
+-- errlike: interval time unit can not be second
+-- Isolates ONLY the interval unit: identical to async-mv-refresh-syntax.pass.sql
+-- except "10 MINUTE" -> "30 SECOND". SKILL.md:95 "Minimum interval: 1 MINUTE";
+-- schema-ddl-gotchas.md gotcha #8 (SECOND not supported).
+CREATE TABLE mvbase_interval (
+ k INT NOT NULL,
+ v INT
+) DUPLICATE KEY(k)
+DISTRIBUTED BY HASH(k) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+CREATE MATERIALIZED VIEW mv_interval_second
+BUILD IMMEDIATE REFRESH AUTO ON SCHEDULE EVERY 30 SECOND
+DISTRIBUTED BY RANDOM BUCKETS 1
+PROPERTIES ("replication_num" = "1")
+AS SELECT k, sum(v) AS sv FROM mvbase_interval GROUP BY k;
diff --git a/verify/ddl/gotchas/bool-default-null.fail.sql b/verify/ddl/gotchas/bool-default-null.fail.sql
new file mode 100644
index 0000000..87693a8
--- /dev/null
+++ b/verify/ddl/gotchas/bool-default-null.fail.sql
@@ -0,0 +1,13 @@
+-- ref: doris-best-practices/references/schema-ddl-gotchas.md:131
+-- desc: BOOLEAN DEFAULT "null" (string "null" as a boolean default) must be rejected — gotcha #11
+-- min_version: 2.0
+-- mode: any
+-- errlike: Invalid BOOLEAN literal: null
+-- Complements bool-default-quoted.fail.sql (which covers the unquoted DEFAULT TRUE
+-- variant). gotcha #11 lists BOTH `DEFAULT TRUE` and `DEFAULT "null"` as hard errors.
+CREATE TABLE t_bool_null (
+ id INT NOT NULL,
+ is_active BOOLEAN DEFAULT "null"
+) DUPLICATE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org