You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "wu-sheng (via GitHub)" <gi...@apache.org> on 2026/10/22 02:45:27 UTC

[PR] Judge whether a conversation achieved its targets, and what it cost [skywalking-ai-sessionizer]

wu-sheng opened a new pull request, #87:
URL: https://github.com/apache/skywalking-ai-sessionizer/pull/87

   This adds the conversation judge for 0.7.0. For each target the person set in a conversation, it says
   whether the target was achieved and how efficiently the time and tokens were spent.
   
   ## What it adds
   
   - **`asz measure`** prints what a conversation's model calls consumed: per talk, with the child agents
     each one started, at any depth, priced where an optional price file names the model. It calls no model.
   - **`asz judge`** runs a plan of stages over one conversation, or a window of it, and writes one
     verified file per stage:
     - `ledger`, computed: every model call counted once, and its owner.
     - `targets`: the targets the person set, and what each input did to them.
     - `achievement`: whether each target was achieved, criterion by criterion, with citations.
     - `subtasks`: for each sub-agent, its outcome, whether its report was true, whether it was used, and
       whether giving it the task was reasonable. A task given wrong (`misdirected`) and an untrue report
       acted on (`misled`) are charged to whoever caused them.
     - `efficiency`: which parts of the agent's own work were avoidable.
     - `close`, computed: each target's final statement, costs, and two scores, achievement and
       efficiency, which are never combined.
   - An evaluator reads the conversation only through a bounded navigation, served as an MCP server to
     agent CLIs and in process to model APIs. Every answer fits a page, every call has a reading budget,
     and nothing past the run's head or window is shown.
   - asz checks every answer against the record. A decision with no valid evidence falls back to the
     value that claims nothing, and the stage file says what was weakened.
   - What judging consumes is counted per model request as the provider reported it, never estimated, and
     printed while a stage runs and when it ends.
   - Judge runs go to the OTLP receiver beside the conversation files. They are removed with their
     session, and scenario removal waits until they are sent.
   - The `sj/1` file format is documented in `docs/en/formats/session-judgment.md`.
   
   ## Docs
   
   - `docs/en/concepts-and-designs/conversation-judge.md`: the design, with the measurements behind it.
   - `docs/en/formats/session-judgment.md`: the stage files.
   - `docs/en/setup/judge.md`, `configuration.md`, `command-line.md`: how to set it up and run it.
   
   ## Tests
   
   `make check` passes: vet, lint, license headers, dependency licenses, and every test under the race
   detector. The runner is tested end to end with stand-in evaluators.
   
   Every guard of the judge was made false in turn, and in the answer checks, the close file and the stage
   file reader and writer, each part of a condition as well; each change was compiled before the tests ran.
   A test fails for every one of them, except a few conditions implied by another and guards that no
   document asz builds can reach, which are kept against a malformed one. The operating system failure paths
   of the evaluators and of the judgment storage are not tested yet; they are listed below.
   
   ## Follow-up
   
   These failure paths return an operating system or transport error, and a test needs a way to make the
   operation fail. Each test should check that the error is returned, and for an evaluator that no request is sent.
   
   - CLI evaluator, `cliExecutor.Run`: writing `prompt.md`, the answer schema and the MCP configuration;
     making Kiro's agent directory and writing its `asz-judge.json`.
   - API evaluator, `apiExecutor.Run` and `apiExecutor.post`: writing `prompt.md`; building the request and
     reading the response body.
   - Bedrock, `credentials` and `callBedrock`: the AWS command-line tool failing, or printing credentials that
     do not decode or lack an access key ID; building the request, the transport, and reading the body.
   - Judgment storage, `JudgmentRuns`, `ReserveRun`, `PublishStage` and `MakeWorkDir`: reading directories,
     taking the root lock, and making the run's directories. Their retirement checks are already tested.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Judge whether a conversation achieved its targets, and what it cost [skywalking-ai-sessionizer]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #87:
URL: https://github.com/apache/skywalking-ai-sessionizer/pull/87#issuecomment-6331731735

   ## Example: what the judge prints
   
   A real run on a local Claude Code conversation that reviewed apache/skywalking#13943, with the build of this pull request. The conversation has 5 main talks, and its longest one started child agents. The evaluators are two agent CLIs, named in `asz.yaml`:
   
   ```yaml
   judge:
     evaluators:
       opus:   {kind: cli, program: claude, model: claude-opus-5-5,   args: ["--effort", "high"]}
       sonnet: {kind: cli, program: claude, model: claude-sonnet-5-5, args: ["--effort", "high"]}
   ```
   
   The price file used here lists neither model, so costs are counted in tokens, and the cost each CLI reported is kept beside them.
   
   ### What the conversation consumed, with no model called
   
   ```
   $ asz measure a412b5c2-b49f-4d4d-8eef-bdc75ff0bfeb
   conversation a412b5c2-b49f-4d4d-8eef-bdc75ff0bfeb  (claude-code/1)
     calls        100, 0 with unknown usage, 2 that never finished, their output unknown
     tokens       in 200  out 107011  cache read 10015728  cache write 673626
     money        unavailable: no call was priced
     unattributed 19 calls of child streams no single talk started
   
     5 main talks; the costliest, with the child agents each started:
            -    58 calls (40 in children)  pr updated, you could review all again.
            -     8 calls (0 in children)  ask 3, and comment for other founds.
            -     5 calls (0 in children)  If module is ON by default, then, it should have fallback l…
            -     7 calls (0 in children)  my mistake, I mean remove branch pr-13943 and your created …
            -     3 calls (0 in children)  reivew branch pr-13943
   ```
   
   ### A judge run
   
   Each stage goes to the evaluator the plan names; here the sub-tasks go to sonnet. What a stage consumes is printed while it runs and when it ends.
   
   ```
   $ asz judge -plan targets=opus,achievement=opus,subtasks=sonnet,efficiency=opus a412b5c2-b49f-4d4d-8eef-bdc75ff0bfeb
   run 7 of a412b5c2-b49f-4d4d-8eef-bdc75ff0bfeb at round 1
     targets by opus, single, 1 calls ...
       targets, 30s: 0 of 1 calls done, 1 running; 2 requests, in 4, cache read 17.4k, cache write 19.5k, out 225 tokens; 73.8k input and 449 output tokens per minute; read 6 KB in 2 calls
     targets complete
       1 calls, 1 attempts; 3 requests, the largest 20.7k of a 1.0M window; in 5.3k, cache read 36.9k, cache write 20.7k, out 2.0k tokens; reported 0.21 USD; read 6 KB in 2 calls; 38s; 99.4k input and 3.2k output tokens per minute
     achievement by opus, stream, 5 calls ...
       achievement, 30s: 1 of 5 calls done, 4 running; 11 requests, in 22, cache read 129.1k, cache write 99.2k, out 2.2k tokens; 456.6k input and 4.3k output tokens per minute; read 63 KB in 17 calls
       achievement, 1m0s: 2 of 5 calls done, 3 running; 26 requests, in 52, cache read 560.8k, cache write 140.6k, out 7.2k tokens; 701.4k input and 7.2k output tokens per minute; read 128 KB in 35 calls
     achievement complete
       5 calls, 5 attempts; 28 requests, the largest 43.9k of a 1.0M window; in 27.9k, cache read 640.4k, cache write 145.2k, out 10.3k tokens; reported 1.50 USD; read 128 KB in 35 calls; 1m8s; 717.8k input and 9.1k output tokens per minute
     subtasks by sonnet, single, 1 calls ...
       subtasks, 30s: 0 of 1 calls done, 1 running; 7 requests, in 14, cache read 188.3k, cache write 53.5k, out 1.3k tokens; 483.5k input and 2.6k output tokens per minute; read 65 KB in 10 calls
     subtasks partial: 1 decisions without valid evidence were weakened
       1 calls, 1 attempts; 8 requests, the largest 54.8k of a 1.0M window; in 6.9k, cache read 241.7k, cache write 54.8k, out 2.4k tokens; reported 0.27 USD; read 65 KB in 10 calls; 36s; 505.6k input and 4.0k output tokens per minute
     efficiency by opus, single, 1 calls ...
       efficiency, 30s: 0 of 1 calls done, 1 running; 2 requests, in 4, cache read 20.3k, cache write 30.4k, out 520 tokens; 101.4k input and 1.0k output tokens per minute; read 26 KB in 4 calls
       efficiency, 1m0s: 0 of 1 calls done, 1 running; 3 requests, in 6, cache read 50.7k, cache write 34.8k, out 1.6k tokens; 85.5k input and 1.5k output tokens per minute; read 39 KB in 6 calls
     efficiency partial: 1 of 6 talks were not labeled
       1 calls, 1 attempts; 4 requests, the largest 42.4k of a 1.0M window; in 8.6k, cache read 85.5k, cache write 42.4k, out 3.4k tokens; reported 0.43 USD; read 39 KB in 6 calls; 1m10s; 117.0k input and 3.0k output tokens per minute
   run 7 judging: in 48.7k, cache read 1.0M, cache write 263.1k, out 18.1k tokens over 3m31.837s; 372.8k input and 5.1k output tokens per minute
   run 7 closed partial: data/a412b5c2-b49f-4d4d-8eef-bdc75ff0bfeb/judgments/j000007
   ```
   
   Two stages are partial, and each says why: one sub-task decision had no valid evidence and was weakened, and one talk of a target was not labeled.
   
   ### The result
   
   Each target the person set, with its outcome, its cost and what its work was; then the sub-tasks, and the two scores, which are never combined. t1 is the request the person corrected right after typing it, so it is replaced and not scored.
   
   ```
   $ asz judge -show 7 a412b5c2-b49f-4d4d-8eef-bdc75ff0bfeb
   run 7  data/a412b5c2-b49f-4d4d-8eef-bdc75ff0bfeb/judgments/j000007
     ledger       complete  by asz, computed
     targets      complete  by opus (claude-opus-5-5), in 5310, out 2001, cache read 36902, cache write 20720, reported 0.21 USD
     achievement  complete  by opus (claude-opus-5-5), in 27941, out 10281, cache read 640433, cache write 145183, reported 1.50 USD
     subtasks     partial   by sonnet (claude-sonnet-5-5), in 6870, out 2388, cache read 241731, cache write 54758, reported 0.27 USD - 1 decisions without valid evidence were weakened
     efficiency   partial   by opus (claude-opus-5-5), in 8593, out 3442, cache read 85459, cache write 42393, reported 0.43 USD - 1 of 6 talks were not labeled
     close        partial   by asz, computed
   
     t1  Review the branch pr-13943 (as first typed; the person later said they meant to remove it).
         not achieved: 0 of 1 criteria met, on no evidence; replaced
         cost 3 calls, 158.5k tokens, observed 0.3 min
         3 calls unpriced
         avoidable 0 tokens and 0.0 min; not reviewed 0 tokens
         work: useful 158.5k
     t2  Remove the local branch pr-13943 and the agent-created branch pr-13943-review.
         achieved: 2 of 2 criteria met, on observed result; closed
         cost 7 calls, 403.9k tokens, observed 0.9 min
         7 calls unpriced
         avoidable 0 tokens and 0.0 min; not reviewed 0 tokens
         work: useful 403.9k
     t3  Review the whole updated PR apache/skywalking#13943 again, judging the default-on/off finding by th…
         achieved: 3 of 3 criteria met, on person accepted; closed
         cost 60.5 calls, 7.3M tokens, observed 18.5 min
         60.5 calls unpriced
         avoidable 101.7k tokens and 2.0 min; not reviewed 290.6k tokens
         work: useful 6.0M; verification 889.4k; avoidable_attempt 101.7k; not_reviewed 290.6k
     t4  Explain why the PR changed GenAIContextResolver, which the person did not know about.
         achieved: 1 of 1 criteria met, on delivered answer; closed
         cost 2.5 calls, 290.5k tokens, observed 1.6 min
         2.5 calls unpriced
         avoidable 0 tokens and 0.0 min; not reviewed 0 tokens
         work: useful 290.6k
     t5  Post on PR 13943 a question to the author about item 3 (the GenAIContextResolver change) and review…
         achieved: 2 of 2 criteria met, on observed result; closed
         cost 8 calls, 1.1M tokens, observed 5.1 min
         8 calls unpriced
         avoidable 0 tokens and 0.0 min; not reviewed 0 tokens
         work: useful 1.1M
   
     sub-tasks      2 of 2 judged: achieved 1, not achieved 1; reports accurate 1, unverifiable 1; use used 1, unknown 1; dispatch reasonable 2
   
     score
       achievement  1.00 over 4 targets judged: achieved 4, partly 0, not achieved 0; 0 cannot be verified; 1 cancelled or replaced, not scored
       efficiency   0.99: avoidable 1% of the reviewed cost, by tokens; 97% of the targets' cost reviewed
       delivered    98% of the targets' cost went to targets achieved, by tokens
   
     conversation   10.8M tokens; 2 calls that never finished, their output not counted
     judging        1.3M tokens (4 stages unpriced); the evaluators reported 2.40 USD
                    over 3.5 min: 372.8k input and 5.1k output tokens per minute
   ```
   
   ### Every run of the conversation
   
   ```
   $ asz judge -list a412b5c2-b49f-4d4d-8eef-bdc75ff0bfeb
   run 1  round 1  2026-10-20T14:15:11Z  complete  targets 5: achieved 4, partly 0, not 1, cannot verify 0
   run 2  round 1  2026-10-20T14:18:04Z  partial  targets 3: achieved 2, partly 0, not 1, cannot verify 0
   run 3  round 1  2026-10-20T14:54:25Z  complete  targets 5: achieved 4, partly 0, not 1, cannot verify 0
   run 4  round 1  2026-10-20T14:57:15Z  partial  targets 3: achieved 1, partly 1, not 1, cannot verify 0
   run 5  round 1  2026-10-20T15:58:46Z  complete  targets 5: achieved 4, partly 0, not 1, cannot verify 0
   run 6  round 1  2026-10-20T16:02:31Z  complete  targets 3: achieved 2, partly 0, not 1, cannot verify 0
   run 7  round 1  2026-10-22T03:01:25Z  partial  targets 5: achieved 4, partly 0, not 1, cannot verify 0
   ```
   
   The run took 3.5 minutes. Writing this example showed two printing faults, both fixed in the commit: a share of a call was cut to two significant digits (60.5 calls printed as 60), and the score line counted a replaced target that the score leaves out.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Judge whether a conversation achieved its targets, and what it cost [skywalking-ai-sessionizer]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #87:
URL: https://github.com/apache/skywalking-ai-sessionizer/pull/87


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org