[jira] [Commented] (CAMEL-25311) camel-wolf-defender: Add a semantic expert for prompt-injection detection

["Claus Ibsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-25311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123470#comment-18123470 ] 

Claus Ibsen commented on CAMEL-25311:
-------------------------------------

Related to observability for a guardrail-type expert like this one, and not a blocker for the initial implementation.

OpenTelemetry is working on guardrail-specific conventions: semantic-conventions-genai PR #427 "run guardrail span and security finding" (still open, so names may change): https://github.com/open-telemetry/semantic-conventions-genai/pull/427

It adds {{gen_ai.run_guardrail.internal}} (in-process checks, which fits a local ONNX expert) and a {{gen_ai.security.finding}} event with:
* {{gen_ai.security.guardrail.*}}: which guardrail ran (the expert and pinned artifact identity);
* {{gen_ai.security.verdict.*}}: what it returned (BENIGN/INJECTION);
* {{gen_ai.security.risk.*}}: classification and score (P(INJECTION); a category such as OWASP LLM01 Prompt Injection);
* {{gen_ai.security.action.type}}: what the *caller* enforced, kept separate from the verdict;
* {{gen_ai.security.policy.*}} / {{policy.rule.id}}: which policy fired.

The verdict/action split matches this issue's rule that the model returns a verdict while block / quarantine / review / continue belongs to Camel routing policy. It also fits the requirement not to log the submitted text: none of these attributes needs the input. Worth keeping in mind for the capability descriptor and result shape, so the expert can emit these once #427 lands.

_Claude Code on behalf of davsclaus_

> camel-wolf-defender: Add a semantic expert for prompt-injection detection
> -------------------------------------------------------------------------
>
>                 Key: CAMEL-25311
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25311
>             Project: Camel
>          Issue Type: New Feature
>          Components: camel-ai
>            Reporter: Luigi De Masi
>            Assignee: Luigi De Masi
>            Priority: Major
>
> h2. Problem and intended behaviour
> Add a dedicated Wolf-Defender semantic expert in {{components/camel-ai/camel-wolf-defender}}, published as {{org.apache.camel:camel-wolf-defender}}, so Camel routes can screen untrusted text for prompt injection through {{camel-semantic}}.
> For example, a route receiving a retrieved document or tool response should be able to evaluate {{ref:injection}} and route the exchange to processing, rejection, or review according to application policy. The route should not need to know the model's label IDs, tokenizer, tensor format, inference library, or document-scoring implementation.
> This is the concrete provider implementation associated with CAMEL-25310, which introduces expert selection, capability reporting, optional instructions, and catalog metadata. Reuse those contracts rather than implementing a separate provider-selection mechanism in this module. Use *expert* in configuration and documentation; implementing the existing {{SemanticAdapter}} SPI does not require renaming that Java API.
> h2. Scope and packaging
> * Create {{camel-wolf-defender}} under the {{camel-ai}} parent and integrate its dependencies, service discovery, generated metadata, documentation, and build registration using Camel conventions.
> * Implement a dedicated semantic adapter/expert for the fixed prompt-injection evaluation.
> * Keep {{camel-semantic}} independent of concrete inference and tokenization libraries. These dependencies belong to the optional Wolf-Defender module.
> * Do not depend on LangChain4j or require a Jev-compatible service. Applications should keep the common semantic contract when combining this expert with other providers.
> * Target local inference for the initial implementation. ONNX Runtime's Java API is a candidate backend; select and document the runtime/tokenizer combination after checking the chosen export. A generic backend framework, remote model-serving protocol, and separate routing endpoint are not prerequisites.
> * Establish a tested baseline with a pinned Wolf-Defender v2 artifact, such as Small. Document the supported model/export/runtime combinations; do not imply that all variants and quantizations have been validated.
> h2. Input and capability contract
> Expose the following capabilities through the framework from CAMEL-25310:
> ||Property||Required contract||
> |Input|Text selected by the semantic declaration's state expression|
> |Result type|BOOLEAN|
> |Positive meaning|Prompt injection or jailbreak-like instruction detected|
> |Probability|Probability assigned to the INJECTION class|
> |Instructions|Unsupported; no question or synthetic instruction prompt is required|
> |Caller-defined criteria|Unsupported|
> |CHOICE / SCORE|Unsupported|
> The dedicated expert supplies the meaning of the evaluation. No new {{task}} keyword is needed, and users do not need to repeat a question such as "Is this a prompt injection?".
> Reject CHOICE/SCORE declarations, unsupported instructions, and arbitrary criteria during declaration validation, before inference. Errors should identify the named evaluation and selected expert. Never ignore unsupported options or silently select a different provider. Validate the selected runtime value as text; structured objects require explicit application selection or conversion rather than an implicit {{toString()}}.
> Preserve the exchange content and existing semantic result publication behaviour. The expert's result should not replace the original message body with a model-specific response.
> h2. Mapping model output to SemanticResult
> The current Wolf-Defender Small model card defines class 0 as BENIGN and class 1 as INJECTION. Its ONNX example returns logits. The adapter must validate the selected artifact's output contract and map the positive class correctly.
> * Convert two-class logits into probabilities using numerically stable softmax, where that is the selected export's contract. Do not apply softmax a second time to an already normalized probability output.
> * Return {{P(INJECTION)}} through the existing BOOLEAN probability representation. For example, a top-label response equivalent to BENIGN with probability 0.97 must yield an injection probability of 0.03, not 0.97.
> * Validate output dimensions, label mapping, finite numeric values, and probability bounds. Incompatible artifacts and malformed outputs are errors.
> * Preserve probability information until the common semantic threshold and uncertainty policy are applied. Do not collapse it to the model's default label first or introduce a conflicting hidden decision threshold.
> * Do not advertise this probability as an application-defined SCORE severity scale, or invent a calibrated confidence guarantee.
> BENIGN means that this evaluation did not detect injection. It does not establish general safety, authorization, or the absence of other threats. Actions such as block, quarantine, review, or continue belong to Camel routing policy; they are not additional model labels.
> h2. Tokenization and document handling
> Own tokenization, input tensors, special tokens, attention masks, padding, and model invocation inside the expert. Use tokenizer assets compatible with the pinned model and validate against reference inference.
> The Small v2 model card describes a 2,048-token window. Its published document evaluation uses overlapping windows with 64-token overlap and normalized Smooth-Max aggregation. A single truncated window does not reproduce that protocol.
> Define and document bounded handling of longer input. Either implement a documented document-scoring policy or reject inputs beyond the supported limit explicitly; never silently discard an unchecked suffix. If adopting the published aggregation, verify the exact formula and parameters against an authoritative implementation rather than guessing from its name. Clearly identify any alternative policy and its implications for threshold selection.
> For supported multi-window evaluation, test injection-bearing content beyond the first window and at window boundaries. Limits should bound both tokenization work and the number of inference windows. Empty, missing, and oversized input must have an explicit, tested outcome.
> h2. Expert configuration and lifecycle
> Keep model-specific configuration on the configured Wolf expert instance: model/tokenizer locations and revision, selected export, runtime options, and input/document limits. Named evaluations retain common semantic options such as state, threshold, and uncertainty policy.
> Support reproducible use of explicitly provisioned local artifacts without requiring downloads during normal route execution. Model weights should not be bundled into the Camel source repository or ordinary module artifact. Document acquisition, artifact identity, and compatible tokenizer/configuration files.
> Load and reuse model resources through Camel-managed lifecycle. Separate declaration/capability inspection from model loading and inference. Close sessions, tensors, and tokenizer/native resources on shutdown and startup failure. Support concurrent evaluations without cross-exchange state leakage, and bound concurrency and resource consumption using the selected runtime's facilities.
> Respect the semantic SPI's timeout, interruption, and shutdown contract. Verify the actual native-runtime cancellation behaviour; a timeout around a Java task must not be described as stopping native inference if it only stops waiting for it.
> Model-loading failures, invalid inputs, inference failures, malformed output, and cancellation must remain errors. They must not become a benign result or an injection probability of zero. Diagnostics should include enough provider/artifact identity to reproduce a problem without logging submitted text or credentials.
> h2. Integration with experts and Camel Catalog
> Register the expert through the discovery/lifecycle mechanism agreed in CAMEL-25310 and allow explicitly configured registry instances. Reuse its explicit/default/sole-expert resolution rules and its error on ambiguity. Two configured Wolf instances may use different artifacts or limits and must retain their separate identities.
> Publish generated static capability metadata for {{wolf-defender}}, linked to {{camel-wolf-defender}}, through the same authoritative definition used by runtime capabilities. Catalog inspection must work without downloading weights or initializing a native runtime. Static metadata describes the provider's contract; it must not claim that an arbitrary local artifact has been validated.
> Support the existing single-evaluation and batch SPI contracts. Framework grouping across experts remains the responsibility of CAMEL-25310. The initial implementation may use the existing sequential batch behaviour; optimized tensor batching is not required. Preserve named result keys and all-or-error publication.
> h2. Illustrative application declaration
> The following uses the syntax proposed in CAMEL-25310 and assumes a configured Wolf expert instance named {{security}}. It is illustrative, pending the final framework API and binding conventions.
> {noformat}
> - semantic:
>     question:
>       injection:
>         expert: security
>         type: boolean
>         state: "${body}"
>         threshold: "{{security.injection.threshold}}"
>         uncertainty: "{{security.injection.uncertainty}}"
>         uncertaintyPolicy: fail
> {noformat}
> Routes use {{ref:injection}} through the semantic language. Provide a runnable example showing expert construction/configuration, artifact provisioning, and routing on the resulting decision. Demonstrate that model-specific options remain outside the named evaluation. Explain how uncertainty and operational errors are handled by the route.
> h2. License and documentation
> The Small model card declares Apache License 2.0 and retains MIT terms for the upstream mmBERT-small portions. Preserve applicable notices for any redistributed material and document model provenance separately from runtime/tokenizer dependency licenses. Confirm the license of the exact selected artifacts as part of implementation.
> Document installation, supported runtime/platform combinations, configuration, fixed BOOLEAN semantics, probability mapping, limits, and error behaviour. Describe false-positive/false-negative limitations without making benchmark or latency claims for an untested Java implementation.
> h2. Validation and acceptance criteria
> # The new {{components/camel-ai/camel-wolf-defender}} module builds and integrates with Camel's normal generated metadata and documentation processes.
> # A named BOOLEAN evaluation without instructions works through {{camel-semantic}}, with both explicit expert selection and the framework's sole-expert discovery behaviour.
> # Unsupported result types, instructions, and criteria fail during declaration validation without running inference. Unknown/ambiguous expert selection follows CAMEL-25310.
> # Tests verify positive-class mapping, stable probability conversion, malformed/non-finite output rejection, and common threshold/uncertainty boundaries, including a high-confidence BENIGN result.
> # Input validation and document limits are explicit and tested. Oversized input is never silently partially screened; any supported windowed policy has boundary and late-window coverage.
> # Lifecycle and failure tests cover resource cleanup, concurrent use, cancellation/timeout behaviour, and errors remaining distinct from negative classifications.
> # Batch evaluation preserves keys, expert-instance identity, and complete result validation without exposing partial success.
> # Static capabilities are discoverable through Camel Catalog, agree with runtime reporting, and can be read without model/native-runtime initialization.
> # Normal unit tests use deterministic fixtures and require neither network access nor a large model download. Add an opt-in integration test using a pinned real artifact to verify tokenizer/inference parity against reference outputs with a documented numeric tolerance.
> # Provide a runnable local example and setup instructions, including the tested artifact/runtime, license/provenance, and representative benign, injection, and difficult-benign inputs. Keep adapter correctness evidence separate from claims about model detection quality.
> h2. Related issue and references
> * [CAMEL-25310: semantic experts, capabilities and catalog metadata|https://issues.apache.org/jira/browse/CAMEL-25310]
> * [Wolf-Defender Small model card and inference examples|https://huggingface.co/patronus-studio/wolf-defender-prompt-injection-small]
> * [Wolf-Defender full model|https://huggingface.co/patronus-studio/wolf-defender-prompt-injection]
> * [Wolf-Defender Small license file|https://huggingface.co/patronus-studio/wolf-defender-prompt-injection-small/blob/main/LICENSE]
> * [Wolf-Defender v2 article|https://patronus.studio/en/posts/wolf-defender-v2-prompt-injection-detection-on-device]
> * [ONNX Runtime Java API|https://onnxruntime.ai/docs/get-started/with-java.html]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)