[I] Epic: drools-journal — durable KieSession via append-only journal [incubator-kie-drools]

["mdproctor (via GitHub)" <gi...@apache.org>]

mdproctor opened a new issue, #6682:
URL: https://github.com/apache/incubator-kie-drools/issues/6682

   Append-only journal for durable `KieSession` survival across JVM restarts. A new `drools-journal` module, parallel to `drools-reliability`, with significantly lower write overhead.
   
   ## Implementation tracks
   
   - [ ] **vol1/Phreak** — primary implementation (this issue)
   - [ ] **vol2 port** — #6682 (depends on vol1 reaching sufficient maturity)
   
   ## Key design points
   
   - Append small typed records (`InsertRecord`, `RetractRecord`, `ModifyRecord`, `RuleMatchRecord`, `SafepointRecord`) rather than serialising full objects
   - DRL `modify` blocks rewritten at compile time to emit `ModifyRecord` (lambda ref + parameters only) — the primary write-efficiency gain
   - Pages are append-only; compaction runs in a separate thread/JVM via a four-step PREPARE/COMMIT protocol — writer never pauses
   - Two `JournalStorage` implementations: Chronicle Queue OSS (single-host default) and Aeron Archive OSS (distributed)
   - Restore is a sequential single-pass scan: rebuild object store → bulk Rete re-propagation → wire TMS → install replay filter
   
   ## Module structure
   
   ```
   drools-journal/
     drools-journal-api/          # JournalStorage SPI, record types, session config, strategy SPIs
     drools-journal-core/         # Session integration: entry point, agenda, compiler hooks, restore
     drools-journal-chronicle/    # Chronicle Queue implementation (default, single-host)
     drools-journal-aeron/        # Aeron Archive implementation (distributed, multi-host)
     drools-journal-tests/        # Integration and correctness tests
   ```
   
   ---
   
   <details>
   <summary>📄 Full Design Spec (2026-04-09)</summary>
   
   ## 1. Problem Statement
   
   `drools-reliability` gives Drools stateful sessions the ability to survive JVM restarts using a
   key-value store backend (Infinispan, H2 MVStore). The approach works but has fundamental performance
   constraints: every fact insert or modify requires writing the full serialised object into the store,
   and the store's random-access semantics add overhead on every operation.
   
   The goal of this design is a **write-optimised durable extension** — a journal — that allows a
   `KieSession` to survive JVM restarts with significantly lower write overhead by:
   
   - Appending small, structured records rather than serialising full objects
   - Capturing object modifications as lambdas + their parameters rather than full object snapshots
   - Organising the log into compactable pages
   - Keeping the write path append-only (no in-place updates, no random seeks)
   - Allowing compaction to run in a separate thread or JVM without blocking the writer
   
   This is a **new module** (`drools-journal`), parallel to `drools-reliability`. The existing
   DB-backed reliability module is unchanged.
   
   ---
   
   ## 2. Module Structure
   
   ```
   drools-journal/
     drools-journal-api/          # JournalStorage SPI, record types, session config, strategy SPIs
     drools-journal-core/         # Session integration: entry point, agenda, compiler hooks, restore
     drools-journal-chronicle/    # Chronicle Queue implementation (default, single-host)
     drools-journal-aeron/        # Aeron Archive implementation (distributed, multi-host)
     drools-journal-tests/        # Integration and correctness tests
   ```
   
   | Module | Responsibility |
   |---|---|
   | `drools-journal-api` | `JournalStorage` SPI, `JournalRecord` sealed hierarchy, `PageRollStrategy` SPI, `ObjectStorageStrategy` SPI, `DurableSessionOption` |
   | `drools-journal-core` | `JournalledNamedEntryPoint`, `JournalledAgenda`, `ReplayFilter`, `ModifyLambdaRegistry`, restore engine, compaction coordinator |
   | `drools-journal-chronicle` | `ChronicleJournalStorage` — Chronicle Queue backed, single-machine |
   | `drools-journal-aeron` | `AeronJournalStorage` — Aeron Archive backed, distributed |
   | `drools-journal-tests` | Shared test infrastructure, correctness and crash-safety tests |
   
   ---
   
   ## 3. Journal Record Types
   
   ```
   JournalRecord (sealed)
     ├── InsertRecord
     ├── RetractRecord
     ├── ModifyRecord
     ├── RuleMatchRecord
     ├── SafepointRecord
     ├── CompactionPrepareRecord
     └── CompactionCommitRecord
   ```
   
   **InsertRecord** — fact inserted; carries `factHandleId`, `logical`, `justifyingRuleMatchId`, and either `EmbeddedPayload` (serialised bytes) or `ExternalRef` (type + DB key).
   
   **RetractRecord** — fact retracted; carries `factHandleId`.
   
   **ModifyRecord** — DRL `modify` block executed; carries `factHandleId`, `lambdaClassRef` (e.g. `Rule_MyRule_modify_0`), and `parameters` (serialised changed values only — the primary write-efficiency gain).
   
   **RuleMatchRecord** — rule activation fired; carries auto-incrementing `id`, `ruleName`, `factHandleIds[]`.
   
   **SafepointRecord** — consistent checkpoint marker; always forces a page roll.
   
   **CompactionPrepareRecord / CompactionCommitRecord** — atomic compaction protocol (see §7).
   
   ---
   
   ## 4. Session Integration
   
   **`DurableSessionOption`** — activates journal mode:
   ```java
   DurableSessionOption.newSession()
       .withObjectStorage(ObjectStorageMode.EMBED)
       .withPageRollStrategy(PageRollStrategies.safepointOnly())
       .withJournalStorage(ChronicleJournalStorage.atPath("/var/drools/journal"))
   ```
   
   **`JournalledNamedEntryPoint`** — extends `NamedEntryPoint`; intercepts all fact operations, appends journal records before delegating to Rete.
   
   **`JournalledAgenda`** — extends `DefaultAgenda`; appends `RuleMatchRecord` after each activation fires; holds current activation ID in thread-local state for logical insert wiring; installs `ReplayFilter` on restore.
   
   **`PageRollStrategy` SPI** — called after every append; built-ins: `SafepointOnly`, `SizeThreshold(n)`, `CountThreshold(n)`, `Composite`. Safepoints always force a roll regardless.
   
   **`ObjectStorageStrategy` SPI** — decides embed vs external reference per object; mirrors `JPAPlaceholderResolverStrategy` in `drools-reliability`.
   
   ---
   
   ## 5. Modify-as-Lambda Compiler Rewrite
   
   In durable mode, the DRL compiler rewrites `modify` blocks:
   
   ```java
   // Normal mode output:
   person.setAge(newAge);
   person.setName(newName);
   update(person);
   
   // Durable mode output:
   person.setAge(newAge);
   person.setName(newName);
   journal.appendModify(factHandle, "Rule_MyRule_modify_0", new Object[]{ newAge, newName });
   update(person);  // Rete notification only — does NOT emit a second InsertRecord
   ```
   
   Lambda class names are deterministic: `Rule_{ruleName}_modify_{index}`. Stable across restarts as long as the rule is unchanged. Plain `update()` outside a `modify` block falls back to full `InsertRecord` snapshot.
   
   `ModifyLambdaRegistry` is populated at `KieBase` build time. On restore, `ModifyRecord` replay applies the lambda directly to the fact in working memory — no Rete notification per modify during restore (bulk re-propagation handles this in Phase 1.5).
   
   If a `lambdaClassRef` cannot be resolved on restore, `JournalSchemaEvolutionException` is thrown. Schema migration tooling is out of scope for v1.
   
   ---
   
   ## 6. JournalStorage SPI and Implementations
   
   ```java
   interface JournalStorage {
       long append(ByteBuffer record);         // returns durable position
       JournalScanner scan(long fromPosition); // sequential scan for restore
       long latestPosition();
       void close();
   }
   ```
   
   **Chronicle Queue (default — single host):** memory-mapped append-only files; sub-µs to 40 µs p99 write latency; 80M+ records/sec; atomic EOF marker per segment for crash safety; multi-JVM via shared mmap; Apache 2.0.
   
   **Aeron Archive (distributed):** UDP-based; ~29 µs round-trip; 20M+ records/sec; native cross-machine replication; SBE record encoding; Apache 2.0.
   
   **InMemoryJournalStorage** (test scope only) — non-durable, in-process, no file or network dependency.
   
   | Dimension | Chronicle Queue OSS | Aeron Archive OSS |
   |---|---|---|
   | Write latency | Sub-µs to 40 µs p99 | ~29 µs round-trip |
   | Throughput | 80M+ records/sec | 20M+ records/sec |
   | Cross-machine | No (commercial only) | Yes — native UDP |
   | Best for | Single host, lowest latency | Multi-host cluster |
   
   ---
   
   ## 7. Compaction Protocol
   
   Compaction merges sparse pages; runs in a separate thread/JVM; writer never pauses. Four-step atomic protocol using journal appends as the coordination medium:
   
   ```
   Step 1 — PREPARE:  append CompactionPrepareRecord { Pm_id, replacedPageIds: [P1, P2] }
   Step 2 — WRITE:    write merged page Pm (writer continues unblocked throughout)
   Step 3 — COMMIT:   append CompactionCommitRecord { Pm_id, replacedPageIds: [P1, P2] }
   Step 4 — RETIRE:   delete P1, P2 from disk (lazy — correctness does not depend on this)
   ```
   
   **Consistency guarantee:** a restoring JVM that sees PREPARE but no COMMIT reads P1 + P2 (ignores Pm). Once COMMIT is seen, Pm is canonical regardless of whether P1/P2 are still on disk.
   
   Sparseness tracked in-memory as `pageId → liveEntryCount`; not persisted (rebuilt cheaply on restore by sequential scan). Page is a compaction candidate when `liveEntryCount / totalEntryCount < 30%` (configurable).
   
   ---
   
   ## 8. Restore Protocol
   
   Sequential single-pass scan — no random access, no separate index file:
   
   ```
   Phase 0  Build compaction state map (PREPARE/COMMIT records → page canonical set)
   Phase 1  Sequential page scan: replay InsertRecord, RetractRecord, ModifyRecord,
            RuleMatchRecord, SafepointRecord into working memory
   Phase 1.5 Bulk Rete re-propagation of all surviving facts in final state
   Phase 2  Wire TMS: restoreLogicalDependency(factHandle, activation) for each logical insert
   Phase 3  Install ReplayFilter (already-fired set) on agenda → session is live
   ```
   
   Rollback to safepoint: records after the last `SafepointRecord` on the final page are discarded. Since safepoints always force a page roll, worst-case data loss is one page since the last safepoint.
   
   ---
   
   ## 9. Decision Log (summary)
   
   | Decision | Choice | Key reason |
   |---|---|---|
   | Journal vs database | Journal | Sequential writes, modify-as-delta, no read-modify-write |
   | New module vs extend StorageManager | New module | `Storage<K,V>` is wrong abstraction for append-only |
   | Modify capture | Compiler-driven + full-snapshot fallback | No proxy overhead; no API change |
   | Page boundary | Pluggable strategy; safepoints always force roll | Flexible + consistent rollback semantics |
   | Object storage | Session default + per-object strategy override | Flexibility without domain model pollution |
   | Compaction threading | Separate thread/JVM, fully non-blocking | Writer never pauses |
   | Page index | Derived on restore by sequential scan | No ongoing write overhead |
   | TMS reconstruction | Surrogate ID on RuleMatchRecord | Compact, causal, no side effects |
   | Storage library | Chronicle Queue (default), Aeron Archive (distributed) | Latency + Apache 2.0 |
   | Compaction atomicity | Four-step PREPARE/COMMIT via journal appends | No filesystem rename tricks; works for both backends |
   
   ---
   
   ## 10. Out of Scope (v1)
   
   - Schema evolution / migration tooling for `lambdaClassRef` changes
   - Encryption of journal pages
   - `drools-journal-aeron` distributed implementation
   - Session replication / active-active clustering
   - Streaming restore
   - Multi-entry-point cross-ordering guarantees
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org