[jira] [Commented] (CAMEL-24698) camel-jbang: follow-ups from the AI agent benchmarks (optional MCP args, CLI validate parity, eval placeholders, validator hints)

["Claus Ibsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-24698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114621#comment-18114621 ] 

Claus Ibsen commented on CAMEL-24698:
-------------------------------------

Item 5, found in the "after" one-shot run on 2026-09-12: the CLI has no way to validate application.properties. camel validate only has the yaml and normalize sub commands, so a file with camel.main.log-level=INFO, camel.main.duration=3600s or camel.main.streamCaching=true (all invented by a local model, all wrong) passes the CLI and fails at startup with PropertyBindingException. The shared camel_validate_source tool from CAMEL-24695 does report all three ("Line 2: log-level Unknown option" ...), so the check exists in camel-jbang-core; camel validate should expose it (camel validate configuration <file>, or validate *.properties given as arguments) so CLI users and tools that shell out to the CLI get the same answer as the MCP tools. Two of the one-shot failures in that run (rest-api, routes) had a valid route and only an invalid properties file.

> camel-jbang: follow-ups from the AI agent benchmarks (optional MCP args, CLI validate parity, eval placeholders, validator hints)
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-24698
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24698
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-jbang, camel-yaml-dsl
>            Reporter: Claus Ibsen
>            Assignee: Claus Ibsen
>            Priority: Major
>
> Follow-ups from benchmarking AI agents (a frontier model and a 22 GB local model) editing Camel integrations through the Camel MCP tools, 2026-09-11/12. The bugs the benchmarks found first are tracked separately (CAMEL-24692, CAMEL-24693, CAMEL-24694, CAMEL-24695); this issue collects the remaining, smaller items so none of them is lost. Each can be fixed on its own.
> h3. 1. Older camel-jbang-mcp wrappers declare every argument required
> 69 of the 82 tools in camel-jbang-mcp mark every parameter as required in their {{@ToolArg}} declarations, so the Quarkus MCP server rejects any call that omits an optional one with "Missing required argument", and the documented defaults cannot be used. Example: {{camel_catalog_component_doc}} with only {{name}} fails with "Missing required argument: component". CAMEL-24695 fixed this for the eleven shared authoring tools and added a test that keeps their flags equal to the shared descriptors; the older wrappers ({{CatalogTools}}, {{DiagnoseTools}} except {{camel_error_diagnose}}, {{RuntimeTools}}, security, migration, OpenAPI, dependency tools) need {{required = false}} on every optional argument. A small model sends only the arguments it needs, so today it cannot use most of these tools.
> h3. 2. camel validate yaml does not check Simple expressions
> The CLI {{camel validate yaml}} validates the YAML DSL schema only. The Camel TUI's save-time validation and the shared {{camel_validate_source}} tool (CAMEL-24695, {{SourceValidator}} in camel-jbang-core) also check endpoint URIs, Simple expressions and {{camel.*}} properties. A frontier model shipped {{${padding(n)}}} (not a Simple function) past the CLI validator; the runtime caught it. {{camel validate yaml}} should use the shared {{SourceValidator}} so all three entry points report the same problems.
> h3. 3. Expression evaluation does not resolve property placeholders
> {{tui_eval_expression}} (now the shared {{camel_eval_expression}}) evaluated {{${body} >= {{hot.threshold}}}} as written and reported a parse error, while the same expression works in a route because placeholders are resolved before Simple is parsed. The evaluator should resolve placeholders from the running integration's properties (or from the project's application.properties when evaluating locally) before evaluating, or say clearly that a placeholder was left unresolved. Re-check after CAMEL-24692.
> h3. 4. Validator error messages should say what to do
> The YAML DSL validator returns the raw schema error. Two cases from the benchmark where the message stopped a small model and would slow a person:
> * "log: property 'loggerName' is not defined in the schema" should add "did you mean logName" (the edit-distance suggestion from CAMEL-24666 already exists in the catalog).
> * "route: property 'onException' is not defined in the schema" should add that onException is a top-level entry, a sibling of route, and show a five-line sample once CAMEL-24693 ships them.
> camel-yaml-dsl-validator already depends on camel-catalog, and the YAML DSL model (CAMEL-24362) knows which entries are top-level and which nest where, so this is a post-processing step over the networknt errors, no schema change.
> h3. Origin
> Rows 12 to 15 of the defects tally kept with the benchmark harness. Related: CAMEL-24692, CAMEL-24693, CAMEL-24694, CAMEL-24695, CAMEL-24666, CAMEL-24362.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)