(causeway) 26/41: CAUSEWAY-4059: proposes entity change evaluation tracing

[da...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

danhaywood pushed a commit to branch CAUSEWAY-4066
in repository https://gitbox.apache.org/repos/asf/causeway.git

commit 1842993089401b1735e5fc5b84ec4a8aa796e0e4
Author: Dan Haywood <da...@haywood-associates.co.uk>
AuthorDate: Thu Sep 24 10:17:32 2026 +0100

    CAUSEWAY-4059: proposes entity change evaluation tracing
---
 .../.openspec.yaml                                 |   2 +
 .../add-entity-change-evaluation-span/design.md    | 107 +++++++++++++++++++++
 .../add-entity-change-evaluation-span/proposal.md  |  32 ++++++
 .../entity-change-evaluation-observation/spec.md   | 100 +++++++++++++++++++
 .../add-entity-change-evaluation-span/tasks.md     |  27 ++++++
 5 files changed, 268 insertions(+)

diff --git a/openspec/changes/add-entity-change-evaluation-span/.openspec.yaml b/openspec/changes/add-entity-change-evaluation-span/.openspec.yaml
new file mode 100644
index 00000000000..75289e4b73f
--- /dev/null
+++ b/openspec/changes/add-entity-change-evaluation-span/.openspec.yaml
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-24
diff --git a/openspec/changes/add-entity-change-evaluation-span/design.md b/openspec/changes/add-entity-change-evaluation-span/design.md
new file mode 100644
index 00000000000..57b553ab7ee
--- /dev/null
+++ b/openspec/changes/add-entity-change-evaluation-span/design.md
@@ -0,0 +1,107 @@
+## Context
+
+`EntityChangeTrackerDefault` enlists candidate property changes throughout a transaction and lazily memoizes their final state.
+At transaction completion, `evaluateChangedProperties()` traverses the enlisted records, obtains each current post-value, compares pre-values and post-values, and retains the records that require publication.
+
+For JDO entities, this traversal can evaluate derived properties as well as persisted state.
+Derived accessors can execute application queries, so the OpenTelemetry Java agent exports their JDBC spans as otherwise unexplained children of the current action between command persistence and audit-trail persistence.
+The work is not owned by the audit extension because it occurs before `EntityPropertyChangePublisher` invokes subscribers.
+
+The tracker catches `ConcurrentModificationException` and repeats the evaluation against a defensive copy.
+Property accessor failures that are recognized as non-deadlocks become an unknown post-value according to existing behavior, while deadlocks and other escaping failures terminate evaluation.
+The lazy result can also be requested through tracker APIs, so the observation must surround the actual evaluation rather than assume `beforeCommit()` is always its first trigger.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Identify the aggregate cost of evaluating enlisted property changes and obtaining current post-values.
+- Make automatic JDBC work from derived property accessors a descendant of a stable semantic span.
+- Preserve one bounded observation around the complete evaluation, including any defensive retry.
+- Avoid empty observations for transactions with no enlisted property-change records.
+- Preserve natural context propagation and all existing entity-change publication semantics.
+- Keep telemetry static and free of domain, identity, payload, and transaction data.
+
+**Non-Goals:**
+
+- Do not add one span per entity, property, accessor, or retry attempt.
+- Do not change which persisted or derived properties are evaluated by JDO or JPA.
+- Do not optimize, cache, suppress, or reorder derived property evaluation.
+- Do not include audit persistence or command persistence in the evaluation span.
+- Do not replace or configure Java-agent JDBC, HTTP, context propagation, sampling, or export behavior.
+- Do not diagnose the application-specific implementation of an expensive derived accessor.
+
+## Decisions
+
+### Observe the actual changed-property evaluation
+
+`EntityChangeTrackerDefault.evaluateChangedProperties()` will own the observation around its complete existing evaluation and defensive-copy retry.
+This is the narrowest aggregate boundary that includes current post-value access and pre/post comparison while excluding publication callbacks.
+
+Wrapping `beforeCommit()` was rejected because it would include entity-change publication and incorrectly make `write audit trail` a child of evaluation.
+Wrapping only the `memoizeChangesIfRequired()` call in `beforeCommit()` was rejected because another tracker API can trigger lazy evaluation earlier, leaving the real work outside that span.
+Wrapping `changedRecords()` was rejected because the concurrent-modification recovery path could create multiple spans for one logical evaluation.
+
+### Use one static aggregate observation
+
+The stable observation name will be `causeway.entitychange.evaluate` and the contextual display name will be `evaluate property changes`.
+The observation provider will use `EntityChangeTrackerDefault` and `CausewayModulePersistenceCommons.NAMESPACE` for the existing static bean and module metadata.
+
+No entity type, property identifier, target, bookmark, pre-value, post-value, user, tenant, transaction identifier, interaction identifier, sequence number, collection size, changed count, or retry count will be attached.
+Per-property spans were rejected because they would amplify span volume, reveal application structure, and potentially encourage attaching sensitive values or identities.
+
+### Emit only when there is candidate work
+
+The tracker will bypass observation creation when the enlisted property-change record collection is empty.
+When records are present, it will create one observation even if comparison ultimately finds that none changed, because post-value evaluation still occurred.
+
+Always emitting a span from `beforeCommit()` was rejected because read-only and unrelated transactions would gain empty framework spans.
+
+### Preserve natural hierarchy and Java-agent ownership
+
+The observation will inherit the Micrometer/OpenTelemetry context current when lazy evaluation begins.
+In the foreground action case, the intended hierarchy is:
+
+[source,text]
+----
+act InvoiceSummary#invoiceAll
+├── UPDATE isiscommand.CommandWithArchive
+├── evaluate property changes
+│   ├── SELECT codaproxy.CodaTax
+│   └── ...
+└── write audit trail
+    ├── INSERT isisaudit.AuditEntryWithArchive
+    └── ...
+----
+
+Evaluation and audit persistence remain siblings because evaluation completes before publication invokes the audit subscriber.
+If evaluation occurs under an interaction, request, or background context without a current action, that natural context remains the parent.
+The Java agent continues to create JDBC spans and automatically adopts the evaluation observation as their current parent.
+
+### Preserve completion and failure semantics
+
+The observation closes synchronously when evaluation returns or throws.
+An escaping exception is recorded on the observation and rethrown unchanged.
+An accessor failure already converted by `PropertyChangeRecord` to an unknown value remains handled and does not become a new escaping failure solely for telemetry.
+The existing `ConcurrentModificationException` retry remains inside the same observation and retains its existing logging and result semantics.
+
+When observation is inactive, the no-op registry executes the same evaluation closure without exporting a Causeway span.
+
+## Risks / Trade-offs
+
+- **[Risk] The span name may suggest that only changed records are accessed even though candidate records are evaluated before filtering.** → Use `evaluate property changes`, which describes the determination process rather than asserting that every candidate changed.
+- **[Risk] Checking for candidate work could race with enlistment.** → Perform the check at the same transaction-scoped evaluation boundary without introducing additional mutable state or changing synchronization behavior.
+- **[Risk] Derived accessors can mutate entities and trigger the existing retry.** → Keep both attempts inside one aggregate observation and preserve the defensive-copy fallback unchanged.
+- **[Risk] A broad transaction callback span could accidentally absorb audit persistence.** → Instrument only actual changed-property evaluation and assert sibling hierarchy in tests.
+- **[Risk] Framework instrumentation could duplicate JDBC spans.** → Add no JDBC instrumentation and verify that Java-agent spans remain unique descendants.
+
+## Migration Plan
+
+No application data or configuration migration is required.
+Deploy the framework update with the existing `observation` profile and OpenTelemetry Java agent configuration.
+Verify representative traces show `evaluate property changes` between command persistence and audit writing, with derived-property JDBC spans below evaluation.
+Rollback consists of reverting the framework instrumentation; entity-change publication and persistence behavior remain compatible throughout.
+
+## Open Questions
+
+None.
diff --git a/openspec/changes/add-entity-change-evaluation-span/proposal.md b/openspec/changes/add-entity-change-evaluation-span/proposal.md
new file mode 100644
index 00000000000..ae62fe0e508
--- /dev/null
+++ b/openspec/changes/add-entity-change-evaluation-span/proposal.md
@@ -0,0 +1,32 @@
+## Why
+
+Entity property-change publication resolves current post-values at transaction completion, and JDO can invoke derived property accessors that issue substantial SQL during this phase.
+Those automatic JDBC spans currently appear as unexplained siblings between command persistence and audit-trail persistence, making their framework purpose and aggregate cost difficult to identify.
+
+## What Changes
+
+- Add one bounded semantic observation around evaluation of enlisted entity property changes and their current post-values.
+- Export stable name `causeway.entitychange.evaluate` with contextual display name `evaluate property changes`.
+- Keep automatic JDBC spans produced while evaluating derived properties as descendants of the aggregate evaluation span.
+- Create no evaluation span when no property-change records are enlisted.
+- Keep entity-change evaluation and audit-trail persistence as sibling phases under their natural current action, interaction, request, or background context.
+- Preserve existing evaluation, retry, exception, transaction, persistence-adapter, Java-agent, no-op registry, sampling, and export behavior.
+- Attach only static framework metadata and exclude entity types, property identifiers, targets, bookmarks, pre-values, post-values, users, tenants, transaction identifiers, interaction identifiers, sequence numbers, and counts.
+
+## Capabilities
+
+### New Capabilities
+
+- `entity-change-evaluation-observation`: Defines the aggregate entity property-change evaluation span, lifecycle, hierarchy, activation, and privacy requirements.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+- Affects transaction-scoped entity change tracking in `persistence/commons`, principally `EntityChangeTrackerDefault`.
+- Adds focused observation lifecycle tests and Java-agent compatibility evidence for automatic JDBC parentage.
+- Exercises existing JDO and JPA property-change publication tests to guard persistence semantics.
+- Updates the tracing operations guide with the transaction-completion hierarchy and privacy boundaries.
+- Introduces no new external dependencies and does not replace Java-agent-owned JDBC instrumentation.
diff --git a/openspec/changes/add-entity-change-evaluation-span/specs/entity-change-evaluation-observation/spec.md b/openspec/changes/add-entity-change-evaluation-span/specs/entity-change-evaluation-observation/spec.md
new file mode 100644
index 00000000000..3e4fe994dfc
--- /dev/null
+++ b/openspec/changes/add-entity-change-evaluation-span/specs/entity-change-evaluation-observation/spec.md
@@ -0,0 +1,100 @@
+## ADDED Requirements
+
+### Requirement: Aggregate entity property-change evaluation observation
+
+When observation is active and entity property-change records are enlisted, the framework SHALL create one `causeway.entitychange.evaluate` observation around the complete evaluation of those records.
+The observation SHALL use contextual display name `evaluate property changes`.
+
+#### Scenario: Candidate property changes are evaluated
+
+- **WHEN** the tracker evaluates one or more enlisted property-change records
+- **THEN** one `causeway.entitychange.evaluate` observation encloses obtaining current post-values and comparing pre-values with post-values
+- **AND** the observation closes after evaluation completes
+
+#### Scenario: Evaluation finds no publishable difference
+
+- **WHEN** one or more candidate records are evaluated but no pre-value and post-value comparison requires publication
+- **THEN** the completed evaluation is still represented by one `causeway.entitychange.evaluate` observation
+
+#### Scenario: No property-change records are enlisted
+
+- **WHEN** transaction completion has no enlisted property-change records to evaluate
+- **THEN** no `causeway.entitychange.evaluate` observation is created
+
+### Requirement: Bounded retry and failure lifecycle
+
+The evaluation observation SHALL preserve the tracker’s existing retry, handled-accessor-failure, and escaping-failure behavior without retaining observation scope beyond the synchronous evaluation.
+
+#### Scenario: Concurrent modification triggers defensive evaluation
+
+- **WHEN** evaluation of the original record collection triggers the existing concurrent-modification recovery path
+- **THEN** the original attempt and defensive-copy retry execute within the same `causeway.entitychange.evaluate` observation
+- **AND** no additional evaluation observation is created for the retry
+
+#### Scenario: Property access failure becomes an unknown value
+
+- **WHEN** existing property-change behavior converts a non-deadlock accessor failure to an unknown post-value
+- **THEN** evaluation continues with unchanged publication semantics
+- **AND** telemetry does not turn the handled accessor failure into a new escaping failure
+
+#### Scenario: Evaluation fails
+
+- **WHEN** entity property-change evaluation throws an escaping exception
+- **THEN** the observation records the failure
+- **AND** the observation scope closes
+- **AND** the same exception is rethrown
+
+### Requirement: Natural parentage and automatic JDBC descendants
+
+The evaluation observation SHALL inherit the current Micrometer/OpenTelemetry context and SHALL make synchronous Java-agent JDBC spans produced by property evaluation its descendants.
+It SHALL complete before entity property-change subscribers are invoked.
+
+#### Scenario: Evaluation occurs during an action transaction
+
+- **WHEN** entity property changes are evaluated while an action context is current
+- **THEN** `evaluate property changes` is a descendant of that action span
+
+#### Scenario: Derived property evaluation executes SQL
+
+- **WHEN** a derived property accessor executes SQL while current post-values are obtained
+- **THEN** the Java-agent JDBC spans are descendants of `causeway.entitychange.evaluate`
+- **AND** the framework does not create duplicate JDBC spans
+
+#### Scenario: Audit persistence follows evaluation
+
+- **WHEN** evaluated property changes are subsequently delivered to the audit-trail subscriber
+- **THEN** `evaluate property changes` closes before `write audit trail` begins
+- **AND** the evaluation and audit observations are siblings under their natural current parent rather than one containing the other
+
+#### Scenario: Evaluation has no current action
+
+- **WHEN** entity property changes are evaluated with an interaction, request, or background context current but no action context current
+- **THEN** the evaluation observation inherits that natural current context
+- **AND** it does not fabricate action parentage
+
+### Requirement: Static metadata and private evaluation telemetry
+
+The evaluation observation SHALL use only static framework metadata needed to identify its bean and module.
+It MUST NOT attach entity types, property identifiers, targets, bookmarks, pre-values, post-values, users, tenants, transaction identifiers, interaction identifiers, sequence numbers, candidate counts, changed counts, retry counts, or other change payload data.
+
+#### Scenario: Evaluation telemetry is inspected
+
+- **WHEN** a `causeway.entitychange.evaluate` observation is exported
+- **THEN** its low-cardinality metadata identifies only the static framework bean and module
+- **AND** its name, contextual name, and attributes contain no entity, property, value, identity, transaction, interaction, sequence, or count data
+
+### Requirement: Inactive observation preserves entity-change behavior
+
+When observation is inactive, entity property-change evaluation and publication SHALL remain unchanged and no Causeway evaluation span SHALL be exported.
+
+#### Scenario: Observation profile is inactive
+
+- **WHEN** enlisted property changes are evaluated without the `observation` profile
+- **THEN** the same post-values are obtained and the same property changes are published
+- **AND** no `causeway.entitychange.evaluate` span is exported
+
+#### Scenario: No-op registry evaluates changes
+
+- **WHEN** the tracker uses a no-op observation registry
+- **THEN** the evaluation closure runs exactly once according to existing retry behavior
+- **AND** no tracing SDK, exporter, or Java-agent replacement is installed by the framework
diff --git a/openspec/changes/add-entity-change-evaluation-span/tasks.md b/openspec/changes/add-entity-change-evaluation-span/tasks.md
new file mode 100644
index 00000000000..c120ede7780
--- /dev/null
+++ b/openspec/changes/add-entity-change-evaluation-span/tasks.md
@@ -0,0 +1,27 @@
+## 1. Entity-Change Evaluation Instrumentation
+
+- [ ] 1.1 Inject the existing `CausewayObservationIntegration` into `EntityChangeTrackerDefault` without adding a tracing SDK, exporter, or persistence dependency.
+- [ ] 1.2 Add stable `causeway.entitychange.evaluate` and contextual `evaluate property changes` observation construction using static persistence-commons module metadata.
+- [ ] 1.3 Wrap the complete `evaluateChangedProperties()` operation so current post-value access, comparison, and any defensive-copy retry share one synchronous observation.
+- [ ] 1.4 Bypass observation creation when no property-change records are enlisted while retaining one observation when candidates are evaluated but none require publication.
+- [ ] 1.5 Preserve handled accessor-failure behavior, record and rethrow escaping failures, and close the observation scope on success and failure.
+- [ ] 1.6 Keep audit publication outside the evaluation observation and attach no entity, property, value, identity, transaction, interaction, sequence, retry, or count metadata.
+
+## 2. Focused and Persistence Tests
+
+- [ ] 2.1 Add focused tracker tests proving one observation encloses post-value evaluation and comparison for one or multiple candidate records.
+- [ ] 2.2 Test that an empty enlisted-record collection creates no evaluation observation and does not alter transaction completion behavior.
+- [ ] 2.3 Test that candidates with no publishable difference still produce one completed evaluation observation.
+- [ ] 2.4 Test that the concurrent-modification recovery path remains within one observation and preserves its defensive-copy retry result.
+- [ ] 2.5 Test current-parent inheritance, synchronous child-operation parentage, success cleanup, escaping-failure recording and rethrow, and handled unknown-value behavior.
+- [ ] 2.6 Test no-op registry behavior and verify that exported names and low-cardinality metadata contain only the stable static bean and module identifiers.
+- [ ] 2.7 Run existing JDO and JPA entity property-change publication and audit-trail persistence tests to confirm adapter, derived-property, ordering, and transaction semantics remain unchanged.
+
+## 3. Java-Agent Compatibility and Operations Evidence
+
+- [ ] 3.1 Extend the Java-agent compatibility fixture with representative aggregate entity-change evaluation that performs multiple automatic JDBC operations before audit writing.
+- [ ] 3.2 Assert `action -> evaluate property changes -> JDBC` ancestry, sibling evaluation and audit spans, one aggregate evaluation span, and no duplicate framework JDBC instrumentation.
+- [ ] 3.3 Verify the compatibility fixture both without the Java agent and with the supported Java 11 and OpenTelemetry Java-agent combination.
+- [ ] 3.4 Update the tracing operations guide with stable and contextual names, transaction-completion hierarchy, JDO-derived-property behavior, activation requirements, and privacy exclusions.
+- [ ] 3.5 Record representative trace evidence showing derived-property SQL grouped beneath `evaluate property changes` and separated from command and audit persistence.
+- [ ] 3.6 Run focused persistence and compatibility test suites, run `git diff --check`, validate the OpenSpec change strictly, and record commands and results in `validation.md`.