You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "Aetherance (via GitHub)" <gi...@apache.org> on 2026/06/20 09:20:25 UTC
[D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
GitHub user Aetherance created a discussion: Proposal: Keyspace Notifications for Kvrocks
## 1. Background and goals
Applications reading from a read-only Kvrocks replica keep a local cache to cut latency and reduce round-trips to the primary. Keeping that cache fresh requires a notification when a key changes. Kvrocks has no such mechanism today, which blocks applications that depend on Redis Keyspace Notifications from migrating.
This design implements the minimal Redis-compatible Keyspace Notifications surface needed by #2915:
- `notify-keyspace-events` configuration.
- Events published to `__keyspace@<db>__:<key>` with payload = event name.
- Events published to `__keyevent@<db>__:<event>` with payload = key.
- Consumption through the existing `SUBSCRIBE` / `PSUBSCRIBE`.
- `SET` and `DEL` notifications on both primary and replica.
The MVP is intentionally narrow. It does not infer full Redis command semantics from RocksDB storage mutations: those records cannot distinguish `SET` from `APPEND`, `SETRANGE`, or `INCR`, nor a genuine `DEL` from lazy expiry or internal delete-and-recreate flows.
## 2. Architecture
Redis emits keyspace events at the command execution point, and replicas re-emit them by replaying the replicated command stream. Kvrocks replicates the RocksDB WAL (`WriteBatch`), not Redis commands, so replicas cannot recover exact command semantics from storage mutations alone.
The design therefore stores explicit keyspace event markers in the same `WriteBatch` as the data mutation:
```mermaid
flowchart TD
subgraph primary["Primary / standalone"]
cmd["Command layer<br/>SET / DEL succeeds"]
marker["Append event marker<br/>to the same WriteBatch"]
write["db_->Write(batch) succeeds"]
decode_primary["Decode markers<br/>from the batch"]
notify_primary["NotifyKeyspaceEvent"]
end
subgraph replica["Replica"]
recv["Receive replicated WriteBatch"]
apply["Apply batch succeeds"]
decode_replica["Decode markers<br/>from the same batch"]
notify_replica["NotifyKeyspaceEvent"]
end
pub["PublishMessage<br/>keyspace / keyevent"]
clients["Local SUBSCRIBE / PSUBSCRIBE subscribers"]
cmd --> marker --> write --> decode_primary --> notify_primary
marker -. "replicated with WriteBatch" .-> recv
recv --> apply --> decode_replica --> notify_replica
notify_primary --> pub
notify_replica --> pub
pub --> clients
```
This keeps the WAL as the replication carrier without guessing event names from column-family writes. The primary publishes after `db_->Write` succeeds; the replica publishes after it applies the same batch. With the same local `notify-keyspace-events` setting, both roles publish the same marker-derived events.
Pub/Sub fan-out is reused unchanged: `Server::PublishMessage` already handles exact-channel and pattern subscriptions and is a pure in-memory operation, so it works on read-only replicas.
## 3. Design decisions
### 3.1 Config: `notify-keyspace-events`
Add the `notify-keyspace-events` config. The default is `""` (disabled). The parser accepts the Redis 7 flag character set `K E A g $ l s h z x e t m n d`, rejects unknown characters, and supports runtime updates through `CONFIG SET`.
The MVP accepts the full flag set for compatibility, but only `g` (`del`) and `$` (`set`) currently produce notifications. Internally these flags are parsed into non-overlapping bits:
| Constant | Bit | Flag | Meaning |
|---|---:|---|---|
| `kKeyspace` | `1 << 0` | `K` | publish keyspace channels |
| `kKeyEvent` | `1 << 1` | `E` | publish keyevent channels |
| `kGeneric` | `1 << 2` | `g` | generic events; MVP emits `del` |
| `kString` | `1 << 3` | `$` | string events; MVP emits `set` |
| `kList` | `1 << 4` | `l` | accepted, no MVP emitter |
| `kSet` | `1 << 5` | `s` | accepted, no MVP emitter |
| `kHash` | `1 << 6` | `h` | accepted, no MVP emitter |
| `kZSet` | `1 << 7` | `z` | accepted, no MVP emitter |
| `kExpired` | `1 << 8` | `x` | accepted, no MVP emitter |
| `kEvicted` | `1 << 9` | `e` | accepted, no MVP emitter |
| `kStream` | `1 << 10` | `t` | accepted, no MVP emitter |
| `kKeyMiss` | `1 << 11` | `m` | accepted, no MVP emitter |
| `kNew` | `1 << 12` | `n` | accepted, no MVP emitter |
| `kModule` | `1 << 13` | `d` | accepted, no MVP emitter |
`A` expands to `g$lshzxetd` for Redis compatibility. It does not include `m` (`keymiss`) or `n` (`new`), which must be enabled explicitly. In this MVP, only `$` and `g` produce events, so the common `KEA` setting enables both channel forms and lets `set` / `del` pass the filter.
The other classes enabled by `A` are accepted but inert for now. If later releases add markers for list/hash/etc., existing `KEA` deployments will naturally start receiving those events; that matches Redis semantics, but should be called out in release notes.
### 3.2 Event marker format
Add a keyspace-event log-data record that can be appended to a RocksDB `WriteBatch`:
```cpp
struct KeyspaceEventLogData {
int type_class; // KeyspaceEvent::kString or KeyspaceEvent::kGeneric
std::string event; // "set" or "del"
std::string ns; // Kvrocks namespace
std::string key; // user key
};
```
The record is encoded with `WriteBatch::PutLogData()` using a versioned, non-numeric binary tag:
```text
"kse1" // 4-byte ASCII tag, leading 'k'
uint8 type_class_code // 1 = kString, 2 = kGeneric
uint8 event_code // 1 = set, 2 = del
uint32 namespace_len_le
bytes namespace
uint32 key_len_le
bytes key
```
`type_class_code` and `event_code` use compact one-byte encodings. The decoder maps them back to the event class and event name before calling the emitter.
Markers use an independent `kse1` tag instead of reusing normal `redis::WriteBatchLogData`. This avoids confusion with existing Redis type logs and lets older readers treat unknown markers as skippable log data rather than command-reconstruction context.
Markers are self-contained: each record carries its own event class, event name, namespace, and key. The decoder does not infer the affected key from neighboring `PutCF` / `DeleteCF` records or depend on marker position inside the batch. Writers may place a marker after the mutation it describes for debuggability, but correctness does not rely on that order.
Unknown versions, unknown codes, truncated payloads, and malformed markers are skipped after at most a warning; they must not fail an already committed write or block replication. During rolling upgrades, older nodes may ignore `kse1` markers, but they must still apply and continue replicating batches that contain them.
Markers are semantic facts, not publication decisions. They are written when a command has definitely performed a supported mutation. Each node still applies its own `notify-keyspace-events` filter when publishing.
Marker write policy:
- If the primary's `notify-keyspace-events` enables no supported class bit (`g` / `$`), supported commands may skip writing markers to avoid default-off overhead. The skip keys only on class bits, never on the `K` / `E` channel selectors, so a primary with a class bit but no selector still writes markers for replicas to publish.
- If the primary skips markers, replicas cannot publish notifications for that write even if their local config is later enabled (the operational consequence for replica caches is stated as a guarantee in §5).
- When enabled, each supported `SET` / deleted key adds one small log-data record to the WAL and replication stream. This is the cost of preserving command semantics for replicas.
Configuration is thus checked twice — at marker-write time on the primary and at publish time on each node; a `CONFIG SET` that disables notifications between the two still ships the marker to replicas but suppresses the primary's local publish.
### 3.3 Marker writers
Only commands with unambiguous MVP semantics write markers:
- `SET`: append a `set` marker only when the command actually writes the key. Conditional variants such as `SET key value NX` must not write a marker when the condition fails.
- `DEL`: append one `del` marker per key actually deleted, each carrying that single key. Missing keys do not produce markers.
Every other write path (the out-of-scope list in §4) writes no markers in this MVP. This is deliberate: emitting no event is safer than emitting a Redis event name that does not match the command semantics.
### 3.4 Event decoder and trigger points
Primary and replica use the same marker decoder, but invoke it at different points:
- The primary decodes the just-written batch after `db_->Write` succeeds, then calls `Server::NotifyKeyspaceEvent`.
- The replica applies the replicated batch first, decodes that same batch, then calls `Server::NotifyKeyspaceEvent`.
The command layer writes markers but never publishes directly. This keeps failed writes silent and prevents a single-node write from publishing once from the command path and once from the batch path.
Publication is bound only to live write/apply paths. During restart, RocksDB replays WAL internally inside `DB::Open`; that recovery does not invoke the notification path, so historical markers are not republished.
Decode errors are non-fatal. A malformed marker is logged and skipped; keyspace notifications must never abort a committed write or stall replication.
### 3.5 Event emitter
`Server::NotifyKeyspaceEvent` applies the local config filter and reuses the existing Pub/Sub path to publish the two Redis-compatible channel forms:
For a `SET foo bar` marker in the default namespace, `KEA` publishes:
```text
__keyspace@0__:foo -> "set"
__keyevent@0__:set -> "foo"
```
For a successful `DEL foo`, it publishes:
```text
__keyspace@0__:foo -> "del"
__keyevent@0__:del -> "foo"
```
### 3.6 `<db>` / namespace mapping
For Redis compatibility, the default namespace maps to db `0`, so standard subscriptions such as `PSUBSCRIBE __keyevent@0__:*` work out of the box.
Non-default Kvrocks namespaces cannot be inserted into `<db>` directly. A namespace literally named `"0"` would collide with Redis db `0` channels and could leak events across tenants. The mapping must therefore be collision-free.
`MapNamespaceToKeyspaceDB(ns)` is defined as:
- default namespace -> `"0"`;
- non-default namespace -> `"ns:" + PercentEncode(ns)`, where percent encoding escapes every byte outside `[A-Za-z0-9_.-]` and also escapes `%`.
Subscribers to `__keyevent@0__:*` therefore only see the default namespace. Subscribers that opt into non-default namespaces use the encoded form, for example `__keyevent@ns:tenantA__:*`. The payload remains only the key name, so namespace identity comes from the channel.
The key component in `__keyspace@<db>__:<key>` and the payload in `__keyevent@<db>__:<event>` are the raw user key bytes, matching Redis behavior. Keys are not percent-encoded, even if they contain `:`, whitespace, or binary bytes.
## 4. Event scope
Supported MVP events:
| Command | Event class | Event | Emitted when |
|---|---|---|---|
| `SET` | `$` | `set` | the command writes the key |
| `DEL` | `g` | `del` | the key is actually deleted |
Out of scope:
- `APPEND`, `SETRANGE`, `GETSET`, `INCR`, and other string commands.
- Hash/list/set/zset/stream element updates.
- `EXPIRE`, `PEXPIRE`, `expired`, and lazy/compaction expiry.
- `FLUSHDB` / `FLUSHALL` and `DeleteRangeCF` notifications.
- `rename_from`, `rename_to`, `new`, `keymiss`, `evicted`, and module events.
`FLUSHDB` is explicitly a limitation for cache invalidation: this MVP does not emit per-key or aggregate flush notifications. Applications that rely on flush visibility need a later design for aggregate `flushdb` / `flushall` events.
## 5. Semantics and guarantees
- Best effort only. Notifications are Pub/Sub messages: they are not persisted, acknowledged, replayed, or retried.
- Failed writes publish nothing. A marker in an uncommitted batch has no visible effect because publication only runs after successful `db_->Write` or replica apply.
- A write commits before its local notification publishes; a crash in that window can make local subscribers miss the event. Replica notifications publish only after apply, so they can lag the primary, and a replica crash between apply and publish loses the event — restart recovery does not replay it, which also prevents duplicate notifications after crashes.
- `notify-keyspace-events` is node-local, and the primary's setting gates marker writing (§3.2). A primary with no supported class bit writes no markers, so its replicas publish nothing no matter how they are configured. For the #2915 replica-cache use case operators must therefore enable a supported class bit (for example `KEA`) on the primary — even if it has no local subscribers — and set the same value on every subscribed replica.
- Rolling upgrades rely on unknown-marker tolerance. Older nodes may ignore the marker and not publish, but they must still apply and replicate the write batch.
- There is no cross-node de-duplication: a client subscribed to both primary and replica may process the same logical mutation twice.
- For one marker, keyspace is published before keyevent on one node. No ordering is promised across different markers — multi-key commands like `DEL k1 k2 k3`, concurrent writes, or events on different nodes. Marker order inside a batch is not part of the contract; each marker is self-contained.
- The event stream is a cache-invalidation hint, not a reliable CDC mechanism.
## 6. Tests
Add focused coverage, grouped by concern.
Configuration:
- Flag parsing, invalid-flag rejection, and `CONFIG SET` runtime changes.
- Bit assignments: `K`, `E`, `g`, `$` map to distinct bits matching marker `type_class` values.
- `A` expansion sets the Redis all-events class bits (`g$lshzxetd`), excludes `m` / `n`, and makes `KEA` pass `set` / `del` through filtering.
Emission:
- `SET` publishes `set` only on success; `NX` / `XX` condition failures publish nothing.
- `DEL` publishes one `del` per actually deleted key, none for missing keys.
- `APPEND`, `SETRANGE`, `INCR`, and expiry paths publish no markers.
- Disabled configuration emits nothing; replica subscribers receive `set` / `del` from master writes.
Markers and replication:
- Write policy: with primary class bits off, supported commands add no marker overhead and replicas have nothing to publish; with them on, markers are present.
- Codec: valid `set` / `del` markers round-trip; unknown versions, unknown codes, and truncated payloads are skipped without aborting `WriteBatch::Iterate`.
- WAL compatibility: a handler that does not recognize `kse1` still iterates and applies a batch containing the marker.
- Self-containment: a marker not adjacent to its mutation still decodes from its own payload.
- A malformed marker on a replica is logged/skipped without blocking replication progress or ack.
Semantics:
- Duplicate prevention: one successful primary write yields exactly one publication, not command-layer plus WAL-replay.
- Crash recovery: WAL recovery republishes no markers; apply-time publication is best-effort and not replayed after restart.
- Runtime race: flags changed by `CONFIG SET` between marker write and publication take effect at publish time.
- Ordering: keyspace precedes keyevent for one marker; no test relies on cross-marker ordering.
- Namespace mapping: default maps to `0`; non-default names are prefixed and percent-encoded, including numeric names like `"0"`, so tenant channels cannot collide.
## 7. References
- [#2915](https://github.com/apache/kvrocks/issues/2915) — feature request driving this design.
- [Redis Keyspace Notifications](https://redis.io/docs/latest/develop/use/keyspace-notifications/) — the compatibility target for channels, payloads, and flags.
GitHub link: https://github.com/apache/kvrocks/discussions/3533
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "Aetherance (via GitHub)" <gi...@apache.org>.
GitHub user Aetherance added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
Thanks @jihuayu. I've reworked the design based on your feedback.
The new approach no longer writes markers into the `WriteBatch` — instead it derives `set`/`del` from records already in the WAL, so nothing extra is added to the WAL or the replication stream. Primary and replica run the same derivation at different points (the primary after its write, the replica reusing the iteration already done in `parseWriteBatch`), so events are identical on both. The scope stays `set`/`del` only, and everything else matches the proposal.
The one behavioral change vs. the proposal: since events are now derived from storage effects rather than written per-command, `set`/`del` are broader than the literal commands:
- `set` fires for any write to a string key — `SET`/`APPEND`/`INCR`/`SETRANGE`/`GETSET`, and even `GETEX` (which only touches the TTL). It can't be narrowed to the literal `SET` command, since the original command isn't recoverable from the WAL.
- `del` fires for any key removal — including a collection being emptied — not just the `DEL` command. This matches Redis ([docs](https://redis.io/docs/latest/develop/pubsub/keyspace-notifications/#events-generated-by-different-commands)).
Is this broadening acceptable for the initial scope? If so, I'll update the proposal accordingly and implement it in two PRs (primary/standalone first, then replica).
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17386894
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "Aetherance (via GitHub)" <gi...@apache.org>.
GitHub user Aetherance edited a discussion: Proposal: Keyspace Notifications for Kvrocks
## 1. Background and goals
Applications reading from a read-only Kvrocks replica keep a local cache to cut latency and reduce round-trips to the primary. Keeping that cache fresh requires a notification when a key changes. Kvrocks has no such mechanism today, which blocks applications that depend on Redis Keyspace Notifications from migrating.
This design implements the minimal Redis-compatible Keyspace Notifications surface needed by #2915:
- `notify-keyspace-events` configuration.
- Events published to `__keyspace@<db>__:<key>` with payload = event name.
- Events published to `__keyevent@<db>__:<event>` with payload = key.
- Consumption through the existing `SUBSCRIBE` / `PSUBSCRIBE`.
- `SET` and `DEL` notifications on both primary and replica.
The MVP is intentionally narrow. It does not infer Redis command semantics from RocksDB storage mutations alone: those records cannot distinguish `SET` from `APPEND`, `SETRANGE`, or `INCR`, nor a genuine `DEL` command from lazy expiry or internal delete-and-recreate flows. Instead, it reuses the existing Redis-type LogData carried in the same write batch to distinguish the supported `set` and `del` cases.
## 2. Architecture
Redis emits keyspace events at the command execution point, and replicas re-emit them by replaying the replicated command stream. Kvrocks replicates the RocksDB WAL (`WriteBatch`), not Redis commands, so replicas need explicit command context when a storage mutation is ambiguous.
The design therefore extends the existing `redis::WriteBatchLogData` context in the same `WriteBatch` as the data mutation:
```mermaid
flowchart TD
subgraph primary["Primary / standalone"]
cmd["Command layer<br/>SET / DEL succeeds"]
logdata["Create existing LogData<br/>for replicas"]
local_event["Keep command-derived<br/>notify fact"]
write["db_->Write(batch) succeeds"]
publish_primary["After write succeeds<br/>publish local fact"]
notify_primary["NotifyKeyspaceEvent"]
end
subgraph replica["Replica"]
recv["Receive replicated WriteBatch"]
apply["Apply batch succeeds"]
decode_replica["Extract notify events<br/>during batch iteration"]
notify_replica["NotifyKeyspaceEvent"]
end
pub["PublishMessage<br/>keyspace / keyevent"]
clients["Local SUBSCRIBE / PSUBSCRIBE subscribers"]
cmd --> logdata --> write --> publish_primary --> notify_primary
cmd --> local_event --> publish_primary
logdata -. "replicated with WriteBatch" .-> recv
recv --> apply --> decode_replica --> notify_replica
notify_primary --> pub
notify_replica --> pub
pub --> clients
```
This keeps the WAL as the replication carrier without guessing event names from column-family writes. The primary already has command execution context, so it keeps the local notification fact and publishes it only after `db_->Write` succeeds. The replica publishes after it applies the same batch and reconstructs the notification from the replicated LogData plus metadata records. With the same local `notify-keyspace-events` setting, both roles publish the same event names and payloads.
Pub/Sub fan-out is reused unchanged: `Server::PublishMessage` already handles exact-channel and pattern subscriptions and is a pure in-memory operation, so it works on read-only replicas.
## 3. Design decisions
### 3.1 Config: `notify-keyspace-events`
Add the `notify-keyspace-events` config. The default is `""` (disabled). The MVP parser accepts the Kvrocks-supported subset of Redis flags `K E A g $ l s h z t`, rejects unsupported or unknown characters, and supports runtime updates through `CONFIG SET`.
Only `g` (`del`) and `$` (`set`) currently produce notifications. The remaining data-type flags are accepted because Kvrocks has those storage types, but they have no MVP emitter yet. Internally these flags are parsed into a new notification bitmask, separate from the existing storage `RedisType` enum:
| Bit | Flag | Meaning |
|---:|---|---|
| `1 << 0` | `K` | publish keyspace channels |
| `1 << 1` | `E` | publish keyevent channels |
| `1 << 2` | `g` | generic events; MVP emits `del` |
| `1 << 3` | `$` | string events; MVP emits `set` |
| `1 << 4` | `l` | accepted, no MVP emitter |
| `1 << 5` | `s` | accepted, no MVP emitter |
| `1 << 6` | `h` | accepted, no MVP emitter |
| `1 << 7` | `z` | accepted, no MVP emitter |
| `1 << 8` | `t` | accepted, no MVP emitter |
In this MVP, `A` expands only to the accepted Kvrocks data-class subset `g$lshzt`. Redis-only or out-of-scope flags such as `a`, `d`, `x`, `e`, `m`, `n`, `o`, and `c` are intentionally not accepted until their event semantics are implemented. Only `$` and `g` produce events, so the common `KEA` setting enables both channel forms and lets `set` / `del` pass the filter.
The other classes enabled by `A` are accepted but inert for now. If later releases add notification extraction for list/hash/etc., existing `KEA` deployments will naturally start receiving those events; that matches Redis semantics, but should be called out in release notes.
### 3.2 Reusing existing LogData
Kvrocks already writes `redis::WriteBatchLogData` into many write batches with `WriteBatch::PutLogData()`. The current source model is:
```cpp
class WriteBatchLogData {
RedisType type_;
std::vector<std::string> args_;
};
```
It is encoded as the numeric Redis type followed by optional arguments. Existing batch iterators already separate server-level log data from Redis-type log data before decoding. This MVP keeps that boundary and only extends Redis-type `WriteBatchLogData`; it does not add a new top-level log-data kind.
The MVP reuses that mechanism and slightly extends the optional arguments with notification command context, for example by adding Redis command codes such as `kRedisCmdSet` and `kRedisCmdDel`. New command codes must be appended without renumbering existing `RedisCommand` values. It does not add an independent `kse1` record and does not put raw key names into LogData arguments. The affected namespace and key are still decoded from the existing metadata column-family `PutCF` / `DeleteCF` records with `ExtractNamespaceKey`, which preserves raw key bytes and avoids adding one log record per key. Primary-subkey and stream records use `InternalKey` elsewhere and are ignored by the `del` extractor.
A new or extended replica-side notification extractor is stateful in the same way as the existing `WriteBatchExtractor`: `LogData` establishes the current Redis type and optional command context, and subsequent metadata-column-family records can produce notifications when both the LogData context and the column-family operation match a supported event.
- `set`: requires a Redis string LogData context explicitly marked as a supported SET command path, then a metadata `PutCF` whose decoded metadata type is `kRedisString`.
- `del`: requires a LogData context explicitly marked as the supported DEL command path, then a metadata `DeleteCF`. It must not fire for primary-subkey deletes such as `HDEL`, `SREM`, `ZREM`, or list element removals.
The LogData must precede the records it describes, matching current write-batch patterns. If the context is missing, malformed, a server log, or unrelated to key notifications, the extractor skips notification emission for those records.
Decode errors are non-fatal in the notification pass. A malformed or unknown LogData command context is logged and skipped for notification purposes; it must not fail an already committed write or block replication.
LogData command context records semantic facts, not publication decisions. They are written only when a command has definitely performed a supported mutation. Each node still applies its own `notify-keyspace-events` filter when publishing.
LogData write policy:
- If the primary's `notify-keyspace-events` enables no supported class bit (`g` / `$`), supported commands may skip adding notification command context to avoid default-off overhead. The skip keys only on class bits, never on the `K` / `E` channel selectors, so a primary with a class bit but no selector still carries the context for replicas to publish.
- If the primary skips notification context, replicas cannot publish notifications for that write even if their local config is later enabled (the operational consequence for replica caches is stated as a guarantee in §5).
- When enabled, each supported batch adds only a small command-context extension to existing Redis LogData; the affected keys come from the normal write records already present in the WAL and replication stream.
Configuration is thus checked twice — at LogData-context write time on the primary and at publish time on each node; a `CONFIG SET` that disables notifications between the two still ships the context to replicas but suppresses the primary's local publish.
### 3.3 Command paths
Only commands with unambiguous MVP semantics add notification command context:
- `SET`: mark the batch as a supported `set` producer only when the literal `SET` command actually writes the key. Conditional variants such as `SET key value NX` must not add notification context when the condition fails. Shared string helpers must not infer `set` from `kRedisString` alone, because other string commands also write string metadata.
- `DEL`: mark the batch as a supported `del` producer only when the literal `DEL` command deletes at least one key. The extractor emits one `del` event per metadata `DeleteCF`, each carrying that single key. Missing keys do not produce write records or notifications. `UNLINK` currently shares `CommandDel` and `MDel`, so the implementation must distinguish the command name or command attributes and avoid enabling `UNLINK` unless it is explicitly added to the MVP scope.
Every other write path (the out-of-scope list in §4) adds no notification command context in this MVP. This is deliberate: emitting no event is safer than emitting a Redis event name that does not match the command semantics.
### 3.4 Event source and trigger points
Primary and replica publish at different points, and only the replica reconstructs notifications from LogData:
- The primary command path keeps a local notification fact for a successful literal `SET` / `DEL`, then calls `Server::NotifyKeyspaceEvent` only after `db_->Write` succeeds. It does not infer primary notifications from LogData or column-family records.
- The replica applies the replicated batch first, then extracts notifications from LogData and metadata records while iterating that same batch, reusing the path already present around `parseWriteBatch`.
Supported command paths pass notification command context to the type/storage operation that creates the existing `WriteBatchLogData` for replicas, but those type/storage operations do not publish. On the primary, publication is a single after-write step from the retained local fact. This keeps failed writes silent and prevents a single-node write from publishing once from the command path and once from the batch path.
Publication is bound only to live write/apply paths. During restart, RocksDB replays WAL internally inside `DB::Open`; that recovery does not invoke the notification path, so historical LogData is not republished.
Decode errors are non-fatal in the notification handler. Malformed or unsupported notification context is logged and skipped; keyspace notifications must never abort a committed write or stall replication.
### 3.5 Event emitter
`Server::NotifyKeyspaceEvent` applies the local config filter and reuses the existing Pub/Sub path to publish the two Redis-compatible channel forms:
For a successful `SET foo bar` in the default namespace, `KEA` publishes:
```text
__keyspace@0__:foo -> "set"
__keyevent@0__:set -> "foo"
```
For a successful `DEL foo`, it publishes:
```text
__keyspace@0__:foo -> "del"
__keyevent@0__:del -> "foo"
```
### 3.6 `<db>` / namespace mapping
For Redis compatibility, the default namespace maps to db `0`, so standard subscriptions such as `PSUBSCRIBE __keyevent@0__:*` work out of the box.
Non-default Kvrocks namespaces cannot be inserted into `<db>` directly. A namespace literally named `"0"` would collide with Redis db `0` channels and could leak events across tenants. The mapping must therefore be collision-free.
`MapNamespaceToKeyspaceDB(ns)` is defined as:
- default namespace -> `"0"`;
- non-default namespace -> `"ns:" + PercentEncode(ns)`, where percent encoding escapes every byte outside `[A-Za-z0-9_.-]` and also escapes `%`.
Subscribers to `__keyevent@0__:*` therefore only see the default namespace. Subscribers that opt into non-default namespaces use the encoded form, for example `__keyevent@ns:tenantA__:*`. The payload remains only the key name, so namespace identity comes from the channel.
The key component in `__keyspace@<db>__:<key>` and the payload in `__keyevent@<db>__:<event>` are the raw user key bytes, matching Redis behavior. Keys are not percent-encoded, even if they contain `:`, whitespace, or binary bytes.
## 4. Event scope
Supported MVP events:
| Command | Event class | Event | Emitted when |
|---|---|---|---|
| `SET` | `$` | `set` | the command writes the key |
| `DEL` | `g` | `del` | the key is actually deleted |
Out of scope:
- `UNLINK`, `APPEND`, `SETRANGE`, `GETSET`, `INCR`, and other string commands.
- Hash/list/set/zset/stream element updates.
- `EXPIRE`, `PEXPIRE`, `expired`, and lazy/compaction expiry.
- `FLUSHDB` / `FLUSHALL` and `DeleteRangeCF` notifications.
- `rename_from`, `rename_to`, `new`, `keymiss`, `evicted`, and module events.
`FLUSHDB` is explicitly a limitation for cache invalidation: this MVP does not emit per-key or aggregate flush notifications. Applications that rely on flush visibility need a later design for aggregate `flushdb` / `flushall` events.
## 5. Semantics and guarantees
- Best effort only. Notifications are Pub/Sub messages: they are not persisted, acknowledged, replayed, or retried.
- Failed writes publish nothing. LogData context in an uncommitted batch has no visible effect because publication only runs after successful `db_->Write` or replica apply.
- A write commits before its local notification publishes; a crash in that window can make local subscribers miss the event. Replica notifications publish only after apply, so they can lag the primary, and a replica crash between apply and publish loses the event — restart recovery does not replay it, which also prevents duplicate notifications after crashes.
- `notify-keyspace-events` is node-local, and the primary's setting gates notification LogData context (§3.2). A primary with no supported class bit writes no notification context, so its replicas publish nothing no matter how they are configured. For the #2915 replica-cache use case operators must therefore enable a supported class bit (for example `KEA`) on the primary — even if it has no local subscribers — and set the same value on every subscribed replica.
- Rolling upgrades rely on older nodes tolerating the slightly extended `WriteBatchLogData` arguments. Older nodes may ignore the notification context and not publish, but they must still apply and replicate the write batch.
- There is no cross-node de-duplication: a client subscribed to both primary and replica may process the same logical mutation twice.
- For one affected key, keyspace is published before keyevent on one node. No ordering is promised across different affected keys — multi-key commands like `DEL k1 k2 k3`, concurrent writes, or events on different nodes. Extraction follows write-batch iteration order, but clients must not depend on cross-key ordering.
- The event stream is a cache-invalidation hint, not a reliable CDC mechanism.
## 6. Tests
Add focused coverage, grouped by concern.
Configuration:
- Flag parsing, rejection of unsupported or invalid flags, and `CONFIG SET` runtime changes.
- Config bitmask: accepted flags map to distinct bits separate from the existing `RedisType` enum, and `g` / `$` make `del` / `set` pass the filter for their event classes.
- `A` expansion sets the accepted Kvrocks data-class bits (`g$lshzt`) and makes `KEA` pass `set` / `del` through filtering.
Emission:
- `SET` publishes `set` only on success; `NX` / `XX` condition failures publish nothing.
- `DEL` publishes one `del` per actually deleted key, none for missing keys.
- `UNLINK`, `APPEND`, `SETRANGE`, `INCR`, and expiry paths publish no notifications.
- Disabled configuration emits nothing; replica subscribers receive `set` / `del` from master writes.
LogData and replication:
- Write policy: with primary class bits off, supported commands add no notification command context and replicas have nothing to publish; with them on, the extended `WriteBatchLogData` context is present.
- Codec: existing `WriteBatchLogData` still decodes Redis type and arguments; new notification command codes for `set` / `del` are recognized, while unknown command context is skipped by the notification handler without aborting `WriteBatch::Iterate`.
- WAL compatibility: handlers that only understand existing Redis type LogData continue to iterate and apply a batch containing the extended arguments.
- Key extraction: notification keys are decoded from metadata-column-family `PutCF` / `DeleteCF` records, not from raw key names stored in LogData arguments.
- A malformed notification context on a replica is logged/skipped without blocking replication progress or ack.
Semantics:
- Duplicate prevention: one successful primary write yields exactly one publication, not command-layer plus WAL-replay.
- Crash recovery: WAL recovery republishes no notifications; apply-time publication is best-effort and not replayed after restart.
- Runtime race: flags changed by `CONFIG SET` between LogData-context write and publication take effect at publish time.
- Ordering: keyspace precedes keyevent for one affected key; no test relies on cross-key ordering.
- Namespace mapping: default maps to `0`; non-default names are prefixed and percent-encoded, including numeric names like `"0"`, so tenant channels cannot collide.
## 7. References
- [#2915](https://github.com/apache/kvrocks/issues/2915) — feature request driving this design.
- [Redis Keyspace Notifications](https://redis.io/docs/latest/develop/use/keyspace-notifications/) — the compatibility target for channels, payloads, and flags.
GitHub link: https://github.com/apache/kvrocks/discussions/3533
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
GitHub user jihuayu added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
Because Kvrocks uses a different expiration mechanism from Redis, I don’t think `expired` events can be implemented correctly on our side. Therefore, I don’t think we need to support this event.
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17421091
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
GitHub user jihuayu edited a comment on the discussion: Proposal: Keyspace Notifications for Kvrocks
Hi @Aetherance.
I’ve been thinking about this for the past few days. I think we can reuse the existing WriteBatchLogData mechanism and extend it slightly. That should allow us to preserve accuracy while keeping the additional WAL overhead relatively small.
However, please don’t implement both the subscription mechanism and the replication/synchronization logic in a single PR. That would make the PR very large and difficult to review. We should start with a smaller scope and break this larger effort into multiple PRs.
In general, a PR with more than about 300 non-test lines of changes is already considered fairly large from a reviewer’s perspective. Larger PRs take substantially more time and effort to review.
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17421066
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "Aetherance (via GitHub)" <gi...@apache.org>.
GitHub user Aetherance added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
> Because Kvrocks uses a different expiration mechanism from Redis, I don’t think `expired` events can be implemented correctly on our side. Therefore, I don’t think we need to support this event.
I agree with this. `expired` should stay out of scope for this proposal. Kvrocks handles expiration differently from Redis, so it’s hard to provide correct Redis-compatible `expired` events, especially on replicas. I’ll keep this MVP focused on `set` and `del` only as requested in issue #2915. Whether to support other commands/events can be discussed separately later.
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17435848
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
GitHub user jihuayu added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
Thanks. I think you can get started. I'll go over the sync stuff tomorrow
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17460646
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
GitHub user jihuayu added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
I think the overall design looks good, but for the first phase, we should only support publishing changes from the primary node and not implement synchronization to replica nodes.
The synchronization logic for replica nodes may need further consideration. I think the current approach is not very efficient from a performance perspective.
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17379423
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "Aetherance (via GitHub)" <gi...@apache.org>.
GitHub user Aetherance added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
Hi @jihuayu , when you have time, I’d appreciate your thoughts on the direction of the proposal. Once the general approach is confirmed, I can revise it into a complete proposal and move this feature forward. Thanks!
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17417225
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "Aetherance (via GitHub)" <gi...@apache.org>.
GitHub user Aetherance added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
@jihuayu I think I'm ready to start implementing. My plan is to split it into two PRs to keep each one easy to review: the first one implementing keyspace notifications on the primary node (the notify-keyspace-events config plus set/del publishing on the `__keyspace@__ / __keyevent@__` channels), and a second one adding replica support on top, deriving the same events from the WAL.
Does that sound reasonable? If so I'll get started on the first PR. Thanks!
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17460612
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "Aetherance (via GitHub)" <gi...@apache.org>.
GitHub user Aetherance added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
Hi @jihuayu , thanks for the suggestion. I agree with the direction of reusing and slightly extending the existing LogData mechanism, and I’ve updated the proposal accordingly.
With the updated design, replicas can use the command context carried in LogData, together with the write batch records, to infer the original command semantics more accurately.
Could you please review the current proposal again when you have time? If it looks reasonable, I’ll start working on the implementation. I’ll split the changes into reasonably scoped PRs to make review easier.
Thanks again for your guidance!
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17435733
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
GitHub user jihuayu edited a comment on the discussion: Proposal: Keyspace Notifications for Kvrocks
Hi @Aetherance.
I’ve been thinking about this for the past few days. I think we can reuse the existing WriteBatchLogData mechanism and extend it slightly. That should allow us to preserve accuracy while keeping the additional WAL overhead relatively small.
However, please don’t implement both the subscription mechanism and the replication/synchronization logic in a single PR. That would make the PR very large and difficult to review. We should start with a smaller scope and break this larger effort into multiple PRs.
In general, a PR with more than about 400 non-test lines of changes is already considered fairly large from a reviewer’s perspective. Larger PRs take substantially more time and effort to review.
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17421066
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Proposal: Keyspace Notifications for Kvrocks [kvrocks]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
GitHub user jihuayu added a comment to the discussion: Proposal: Keyspace Notifications for Kvrocks
Hi @Aetherance.
I’ve been thinking about this for the past few days. I think we can reuse the existing WriteBatchLogData mechanism and extend it slightly. That should allow us to preserve accuracy while keeping the additional WAL overhead relatively small.
However, please don’t implement both the subscription mechanism and the replication/synchronization logic in a single PR. That would make the PR very large and difficult to review. We should start with a smaller scope and break this larger effort into multiple PRs.
In general, a PR with more than about 300 lines of changes is already considered fairly large from a reviewer’s perspective. The review effort and time required increase significantly as the size grows.
GitHub link: https://github.com/apache/kvrocks/discussions/3533#discussioncomment-17421066
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org