[jira] [Created] (CAMEL-25738) camel-test-dst - deterministic simulation testing for Camel consumers (stage 1)

["Claus Ibsen (Jira)" <ji...@apache.org>]

Claus Ibsen created CAMEL-25738:
-----------------------------------

             Summary: camel-test-dst - deterministic simulation testing for Camel consumers (stage 1)
                 Key: CAMEL-25738
                 URL: https://issues.apache.org/jira/browse/CAMEL-25738
             Project: Camel
          Issue Type: New Feature
          Components: camel-test
            Reporter: Claus Ibsen
            Assignee: shashank
             Fix For: 4.24.0


A new {{camel-test-dst}} module for deterministic simulation testing of Camel consumers: one seeded scheduler runs every Camel thread pool on the test thread, time is virtual, a fake of the external system injects faults chosen by the seed, invariants are checked after each run, and a failing seed replays with the same trace.

The prototype is on the branch {{camel-test-dst-prototype}} of github.com/allthingssecurity/camel ({{components/camel-test/camel-test-dst}}). This issue is the scope of stage 1: making the harness trustworthy enough that a fake world can be written for a component without running into the harness. The chaos proposal waits until this is done.

h3. Evidence so far

A third world was written for camel-aws2-sqs (the visibility timeout extender, a fake SQS queue written from the AWS API reference, 458 lines):
* main: 0 of 5000 seeds fail; {{Sqs2Consumer}} before CAMEL-25700: 1383 of 5000 fail, only with {{maxMessagesPerPoll >= 2}} and {{concurrentConsumers >= 2}}, as that fix describes; a failing seed replays with the same SHA-256.
* it found CAMEL-25737 ({{visibilityTimeout=1}}), because the seed also picks the endpoint configuration.
* it needed a harness fix first (item 1): without it, 271 of 500 seeds on main were false failures.

h3. Stage 1 scope

# *Time read outside the instrumented package.* Byte code is rewritten only in the package of {{@DstInstrument}}, and {{java.time}} was not rewritten at all ({{Instant.now()}} in the SQS extender). Core clocks still read real time: {{MonotonicClock}} (camel-support), {{StopWatch}} (camel-util), {{ScheduledPollConsumer}}. Decide between rewriting the core packages too and a small time-source hook in core that DST replaces (the README suggests the hook; it touches core, so it needs a dev@ discussion).
# *Real sleeps on the loop thread.* {{RedeliveryPolicy.sleep()}} ({{Thread.sleep}}, used by the default synchronous redelivery, {{asyncDelayedRedelivery=false}}), {{ForegroundTask}} (used by 10 component source files), {{DefaultShutdownStrategy}} (waits for in-flight exchanges with {{Thread.sleep}}; the Slack world copies the suspend sequence by hand because of it). A virtual sleep that becomes {{dst.block()}} on the loop thread.
# *Timed waits never time out.* {{DstTask.get(timeout)}} ignores the timeout and {{awaitTermination}} returns true, so timeout paths (circuit breaker, shutdown, {{Future.get(timeout)}}) cannot be explored. Timeouts must fire in virtual time.
# *Waits on something that is not a DST task deadlock.* {{CompletableFuture.get()}}/{{join()}} or {{CountDownLatch.await()}} on the loop thread hang (the SQS consumer with more than 10 messages per poll). At least fail fast with a clear message; better, rewrite them to run the loop until the future completes.
# *Escapes fail the seed, not only the report.* The {{Instant.now()}} gap showed up only as a line in {{escapes.txt}}; a mode where reaching an uncontrolled escape at runtime fails the seed makes the cause obvious.
# *Module layout.* camel-test-dst holds the harness only; the demos (azure-eventhubs, slack, and the sqs world) move to the tests of their component modules as passing regression tests. Refresh the README numbers: the fixes of #27922 and #27923 are on main.
# *A guide for AI agents (and humans) on writing a fake world*, in the module doc page with a pointer in {{AGENTS.md}}, using the SQS world as the worked example:
#* find where the client is swapped (an endpoint option such as {{amazonSQSClient}}, else a static mock of the factory);
#* run the escape scan before writing anything, and fix, avoid or document every escape;
#* write the fake from the service's documentation, quoting the rule for each behaviour and marking assumptions;
#* the one-stack timing rule: a blocked task resumes only after the tasks that blocked after it, so reads take effect at the end of their latency, writes at the start, and background tasks (extenders, timers) do not block;
#* let the seed pick the endpoint configuration as well as the timing;
#* seeded faults, switched off for a drain phase, with a record of which injected fault excuses which violation;
#* the invariant from the component's contract, starting from the reusable {{Invariants}};
#* the self-check: fail on the code before a past fix ({{git log}}), pass on main, and the same SHA-256 when a seed is replayed;
#* triage before reporting: group failing seeds by configuration, read one trace, tell harness artifact, fake mistake and real bug apart.

Each harness fix comes with a self-test, like {{DstSchedulerTest}}.

h3. Later stages (not in this issue)

* The one-stack model itself (a real fix may need virtual threads with a custom scheduler); stage 1 documents it.
* Async clients on Reactor's own schedulers (the Azure SDKs).
* Several runs in parallel in one JVM (the active run is a static field).
* Sharing the fault rules with the chaos proposal.

_Claude Code on behalf of Claus Ibsen_



--
This message was sent by Atlassian Jira
(v8.20.10#820010)