You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "Totoro-qaq (via GitHub)" <gi...@apache.org> on 2026/10/13 07:12:44 UTC

[I] Goal evaluator cannot see tool results or human card answers [maka]

Totoro-qaq opened a new issue, #6086:
URL: https://github.com/apache/maka/issues/6086

   ## What happened
   
   The Host Goal evaluator receives only the last six non-empty user/assistant text messages. Canonical tool calls and results are filtered out, including the actual answers returned by `AskUserQuestion`.
   
   Ordinary Runtime forms have a related gap: accepted values and cancellation/decline are persisted in the Interaction authority, but a form acknowledgement carries only identity. The eventual MCP/plugin tool result does not have to echo the human's response.
   
   This makes materially different execution evidence indistinguishable to the judge. With identical assistant prose, a passing test result and a failing result produce the same evaluation context; answering Yes and No to the same approval card also produces the same context. This can contribute to incorrect completion, progress or waiting judgments. The omission is deterministic; the resulting model judgment depends on the model and task.
   
   ## How to reproduce
   
   1. Arm a Goal that requires passing tests and human approval before publishing.
   2. Complete a Turn containing a Bash call/result, or an `AskUserQuestion` call with an actual human response.
   3. Capture the Goal evaluator request. Its recent context contains the user text and assistant report, but neither the command's result nor the chosen answer.
   4. Keep the prose unchanged and change the result from PASS to FAIL, or the answer from Yes to No. The evaluator context remains identical.
   
   The accompanying local regression uses a real SQLite Session and `HostGoalCoordinator`, with a recording evaluator and no provider calls. All three initial contrasts fail against the current text-only implementation. A separate canonical Interaction fixture covers forms whose tool output does not echo their answers.
   
   ## Environment
   
   - Baseline: `apache/maka` main at `32b1750487ad2ccee6da5928c5dc8f98851f5bca`.
   - Surface: Runtime Host Goal evaluation, shared by Desktop/TUI/CLI.
   - Local validation: macOS, Node.js 24.18.0, npm 11.19.0.
   
   ## Additional context
   
   The text-only filter is in [`goal-coordinator.ts`](https://github.com/apache/maka/blob/32b1750487ad2ccee6da5928c5dc8f98851f5bca/packages/runtime-host/src/server/goal-coordinator.ts#L752-L763). #5768 already preserves useful text rows and report endings; this issue concerns the evidence types that are still omitted.
   
   Proposed fix: retain the existing six prose excerpts, add bounded/redacted canonical tool evidence, and join recent Runtime form acknowledgements to their matching Interaction outcomes. Preserve call/Turn and interaction identities, selected option semantics, error/status/exit-code evidence, and explicit unavailable/truncation markers. Avoid loading archives, raw provider replay, thinking or media bytes.
   
   The scope is evaluator input visibility. It does not change the judgment rubric, continuation strategy or Goal budgets, and does not make the judge an artifact verifier.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org


Re: [I] Goal evaluator cannot see tool results or human card answers [maka]

Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han closed issue #6086: Goal evaluator cannot see tool results or human card answers
URL: https://github.com/apache/maka/issues/6086


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org


Re: [I] Goal evaluator cannot see tool results or human card answers [maka]

Posted by "Totoro-qaq (via GitHub)" <gi...@apache.org>.
Totoro-qaq commented on issue #6086:
URL: https://github.com/apache/maka/issues/6086#issuecomment-6140329618

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org