You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by pi...@apache.org on 2026/08/19 09:07:35 UTC
(iggy-website) branch main updated: fix(site): correct stale claims, fix docs layout and navigation (#64)
This is an automated email from the ASF dual-hosted git repository.
spetz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iggy-website.git
The following commit(s) were added to refs/heads/main by this push:
new f205971e fix(site): correct stale claims, fix docs layout and navigation (#64)
f205971e is described below
commit f205971e98a504bd19c35c6d09359df1f8cef75d
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Aug 19 11:07:31 2026 +0200
fix(site): correct stale claims, fix docs layout and navigation (#64)
The homepage and the docs disagreed with the server they describe.
The hero stat said 7 language SDKs while 8 exist, C++ was listed as
coming soon although that SDK shipped, and the clustering banner
pointed at a server-ng module that no longer exists. VSR clustering
is built into the server and enabled through the [cluster] config
section. The docs index also implied HTTP speaks the custom binary
protocol, when HTTP is REST.
The docs further contradicted themselves about who picks a
partition: some pages said the server round-robins and hashes
message keys, the C# pages claimed the broker never routes. The
binary SDKs resolve balanced and key partitioning client-side and
send an explicit partition id, while the server keeps a live
admission fallback that the HTTP API and the Node SDK rely on. Both
rules are now stated, with the SDK partition-count cache caveat.
The connector observability page listed the counter metrics with a
doubled _total_total suffix, plus a paragraph explaining that the
encoder appends its own suffix. The runtime is being fixed to
register those counters without the literal suffix, so the page now
documents a single _total and drops the explanation. This part only
becomes true once that upstream fix ships.
Presentation lagged too. On phones the sticky docs subnav is 80
percent opaque, so bright hero chips scrolling under it read as
content leaking into the header, and the hero stat row and Why Iggy
table kept their desktop column counts, wrapping one word per line
below 430px. Page names such as Introduction repeat across chapters,
so a bare prev or next card did not say where the arrow led. The
mobile subnav is now opaque like the desktop header, both grids
collapse on small screens, and each card carries the target's
nearest parent folder name, for example Server: Introduction.
---
content/docs/ai/mcp.mdx | 4 +-
content/docs/cli/commands.mdx | 8 ++--
content/docs/cli/start.mdx | 14 +++---
content/docs/clustering/client-failover.mdx | 6 +--
content/docs/clustering/configuration.mdx | 10 ++--
content/docs/clustering/deploy.mdx | 12 ++---
content/docs/clustering/security.mdx | 10 ++--
content/docs/clustering/vsr.mdx | 6 +--
content/docs/connectors/introduction.mdx | 6 +--
content/docs/connectors/observability.mdx | 18 ++++----
content/docs/connectors/runtime.mdx | 2 +-
content/docs/connectors/sdk.mdx | 12 ++---
content/docs/connectors/sinks/clickhouse.mdx | 8 ++--
content/docs/connectors/sinks/delta.mdx | 10 ++--
content/docs/connectors/sinks/doris.mdx | 8 ++--
content/docs/connectors/sinks/elasticsearch.mdx | 4 +-
content/docs/connectors/sinks/http.mdx | 20 ++++----
content/docs/connectors/sinks/iceberg.mdx | 4 +-
content/docs/connectors/sinks/influxdb.mdx | 4 +-
content/docs/connectors/sinks/meilisearch.mdx | 6 +--
content/docs/connectors/sinks/s3.mdx | 2 +-
content/docs/connectors/sinks/surrealdb.mdx | 4 +-
content/docs/connectors/sources/elasticsearch.mdx | 4 +-
content/docs/connectors/sources/influxdb.mdx | 14 +++---
content/docs/connectors/sources/postgres.mdx | 6 +--
content/docs/connectors/sources/source.mdx | 12 ++---
content/docs/connectors/transforms.mdx | 14 +++---
content/docs/faq/faq.mdx | 4 +-
content/docs/index.mdx | 2 +-
content/docs/introduction/about.mdx | 2 +-
content/docs/introduction/architecture.mdx | 4 +-
content/docs/introduction/concepts.mdx | 20 ++++----
content/docs/introduction/getting-started.mdx | 10 ++--
content/docs/sdk/connection-strings.mdx | 4 +-
content/docs/sdk/cpp/intro.mdx | 6 +--
content/docs/sdk/csharp/examples.mdx | 2 +-
content/docs/sdk/csharp/guide.mdx | 8 ++--
content/docs/sdk/csharp/high-level-sdk.mdx | 2 +-
content/docs/sdk/csharp/intro.mdx | 16 +++----
content/docs/sdk/go/examples.mdx | 4 +-
content/docs/sdk/go/intro.mdx | 12 ++---
content/docs/sdk/introduction.mdx | 4 +-
content/docs/sdk/java/examples.mdx | 4 +-
content/docs/sdk/java/intro.mdx | 2 +-
content/docs/sdk/node/examples.mdx | 2 +-
content/docs/sdk/node/intro.mdx | 12 +++--
content/docs/sdk/php/intro.mdx | 18 ++++----
content/docs/sdk/python/examples.mdx | 2 +-
content/docs/sdk/python/intro.mdx | 10 ++--
content/docs/sdk/rust/examples.mdx | 6 +--
content/docs/sdk/rust/high-level-sdk.mdx | 16 +++----
content/docs/sdk/rust/intro.mdx | 4 +-
content/docs/sdk/rust/stream-builder.mdx | 15 +++---
content/docs/server/benchmarking.mdx | 10 ++--
content/docs/server/configuration.mdx | 36 +++++++--------
content/docs/server/docker.mdx | 6 +--
content/docs/server/introduction.mdx | 12 ++---
content/docs/server/networking.mdx | 16 +++----
content/docs/server/schema.mdx | 56 ++++++++++++-----------
content/docs/server/security.mdx | 10 ++--
content/docs/server/storage-engine.mdx | 26 +++++------
content/docs/server/topic-options.mdx | 16 +++----
content/docs/web_ui/start.mdx | 10 ++--
src/app/(home)/page.tsx | 25 +++++-----
src/app/docs/[[...slug]]/page.tsx | 35 +++++++++++++-
src/app/global.css | 6 +++
src/components/architecture-diagrams.tsx | 10 ++--
src/components/code-tabs.tsx | 4 +-
68 files changed, 370 insertions(+), 327 deletions(-)
diff --git a/content/docs/ai/mcp.mdx b/content/docs/ai/mcp.mdx
index b402db34..c53922bd 100644
--- a/content/docs/ai/mcp.mdx
+++ b/content/docs/ai/mcp.mdx
@@ -12,7 +12,7 @@ The [docker image](https://hub.docker.com/r/apache/iggy-mcp) is available, and c
## Configuration
-The minimal viable configuration requires at least the Iggy credentials, to create the connection with the running Iggy server using TCP with which the MCP server will communicate. You can choose between HTTP (the default) and STDIO transports (e.g. for the local usage with tools such as [Claude Desktop](https://claude.ai/download) choose `stdio`).
+The minimal viable configuration requires at least the Iggy credentials, used to create the TCP connection to the running Iggy server that the MCP server talks to. You can choose between HTTP (the default) and STDIO transports (e.g. for the local usage with tools such as [Claude Desktop](https://claude.ai/download) choose `stdio`).
```toml
transport = "http" # http (default) or stdio are supported
@@ -67,7 +67,7 @@ transport = "grpc" # grpc or http
endpoint = "http://localhost:4317"
```
-The configuration file must be in the `toml` format. The path to the configuration can be overridden by `IGGY_MCP_CONFIG_PATH` environment variable. Each setting can be also additionally overridden by using the following convention `IGGY_MCP_<SECTION>_<KEY>` e.g. `IGGY_MCP_IGGY_USERNAME`, `IGGY_MCP_HTTP_ADDRESS` and so on. Environment variables can also be loaded from a dotenv file: point `IGGY_MCP_ENV_PATH` at the file, otherwise a `.env` file in the current working directory is loaded [...]
+The configuration file must be in the `toml` format. The path to the configuration can be overridden by `IGGY_MCP_CONFIG_PATH` environment variable. Each setting can also be overridden by using the following convention `IGGY_MCP_<SECTION>_<KEY>` e.g. `IGGY_MCP_IGGY_USERNAME`, `IGGY_MCP_HTTP_ADDRESS` and so on. Environment variables can also be loaded from a dotenv file: point `IGGY_MCP_ENV_PATH` at the file, otherwise a `.env` file in the current working directory is loaded automatically.
The `token` value can be either a literal PAT or a `file:` reference such as `token = "file:/run/secrets/iggy_pat"`, in which case the token is read from the given file (`~` is expanded to the home directory).
diff --git a/content/docs/cli/commands.mdx b/content/docs/cli/commands.mdx
index 6c483926..9b91b710 100644
--- a/content/docs/cli/commands.mdx
+++ b/content/docs/cli/commands.mdx
@@ -3,7 +3,7 @@ title: Command Reference
description: Every iggy CLI command with its arguments, flags, and a runnable example
---
-This page lists every top-level `iggy` command. Examples assume you are already authenticated, for example via `iggy login`; see [installation and authentication](/docs/cli/start). Run `iggy <command> --help` for the full flag list of any command.
+This page lists every top-level `iggy` command. Examples assume you're already authenticated, for example via `iggy login`. See [installation and authentication](/docs/cli/start). Run `iggy <command> --help` for the full flag list of any command.
{/* Keep this block in sync with the Command enum in core/cli/src/args/mod.rs. */}
@@ -39,7 +39,7 @@ Conventions used throughout:
- Wherever a command takes a stream, topic, user, or consumer group ID, you can pass either the numeric ID or the name (`iggy topic get dev events` and `iggy topic get 1 1` are equivalent).
- Subcommands have single-letter aliases too: `c` (create), `d` (delete), `g` (get), `l` (list), `u` (update), `p` (purge). So `iggy s l` is `iggy stream list`.
- Every `list` command accepts `-l, --list-mode <table|list>` (default: `table`).
-- `-q, --quiet` suppresses stdout output; `-d, --debug <file>` writes verbose logs to a file.
+- `-q, --quiet` suppresses stdout output. `-d, --debug <file>` writes verbose logs to a file.
## stream
@@ -70,7 +70,7 @@ iggy topic create [OPTIONS] <STREAM_ID> <NAME> <PARTITIONS_COUNT> <COMPRESSION_A
iggy topic update [OPTIONS] <STREAM_ID> <TOPIC_ID> <NAME> <COMPRESSION_ALGORITHM> [MESSAGE_EXPIRY]...
```
-Compression is a **required positional**: `none` or `gzip`. Expiry is a human-readable duration (`7d`, `1day 12h`, `unlimited`); omitting it uses the server default.
+Compression is a **required positional**: `none` or `gzip`. Expiry is a human-readable duration (`7d`, `1day 12h`, `unlimited`). Omitting it uses the server default.
| Flag | Description |
|------|-------------|
@@ -129,7 +129,7 @@ iggy message send [OPTIONS] <STREAM_ID> <TOPIC_ID> [MESSAGES]...
| Flag | Description |
|------|-------------|
| `-p, --partition-id <ID>` | Send to a specific partition |
-| `-m, --message-key <KEY>` | Let the server pick the partition from the key (mutually exclusive with `--partition-id`) |
+| `-m, --message-key <KEY>` | Route by message key, hashed to a partition client-side (mutually exclusive with `--partition-id`) |
| `-H, --headers <KEY:KIND:VALUE>` | Message headers, comma separated. Kinds: `raw`, `string`, `bool`, `int8`-`int128`, `uint8`-`uint128`, `float32`, `float64` |
| `--input-file <FILE>` | Send messages from a binary file written by `poll --output-file` |
diff --git a/content/docs/cli/start.mdx b/content/docs/cli/start.mdx
index 04175300..327ca1d7 100644
--- a/content/docs/cli/start.mdx
+++ b/content/docs/cli/start.mdx
@@ -2,7 +2,7 @@
title: Iggy CLI
---
-The Iggy CLI is a command-line interface for managing an Iggy server: streams, topics, partitions, messages, users, tokens and more. It lives in the core repository under the `core/cli` directory. The crate is named `iggy-cli`, while the installed binary is named `iggy`.
+The Iggy CLI is a command-line interface for managing an Iggy server: streams, topics, partitions, messages, users, tokens and more. It lives in the core repository under the `core/cli` directory. The crate is named `iggy-cli`, the installed binary `iggy`.
This page covers installing the CLI, connecting to a server, authenticating, and managing connection contexts. The full command tree is documented in the [command reference](/docs/cli/commands).
@@ -32,7 +32,7 @@ docker run --rm -it --network host --entrypoint iggy apache/iggy -u iggy -p iggy
`--network host` works on **Linux**. On macOS and Windows, point the CLI at the host instead: `--tcp-server-address host.docker.internal:8090`.
-Once installed, run `iggy` without arguments to print an overview of the available commands, or head to the [command reference](/docs/cli/commands).
+Once installed, run `iggy` without arguments to print an overview of the available commands, or see the [command reference](/docs/cli/commands).
## Connecting to a server
@@ -77,7 +77,7 @@ Commands that talk to the server require credentials. The CLI resolves them in t
4. `-u, --username <user>` with `-p, --password <password>`. When `-p` is omitted, the CLI prompts for the password interactively (or reads one line from stdin when piped).
5. The `IGGY_USERNAME` and `IGGY_PASSWORD` environment variables (both must be set).
-`-u`, `-t` and `-n` are **mutually exclusive**. Avoid `-p` with the password inline where possible: it lands in your **shell history**. Prefer `iggy login`, the interactive prompt, or the environment variables.
+`-u`, `-t` and `-n` are **mutually exclusive**. Avoid passing the password inline with `-p`: it lands in your **shell history**. Prefer `iggy login`, the interactive prompt, or the environment variables.
```bash
# Interactive password prompt
@@ -107,7 +107,7 @@ iggy session status
iggy logout
```
-`iggy session status` only checks whether a token exists in the local keyring; an expired token **still reports as active**. Run `iggy me` to verify the session against the server.
+`iggy session status` only checks whether a token exists in the local keyring. An expired token **still reports as active**. Run `iggy me` to verify the session against the server.
### Personal access tokens
@@ -121,7 +121,7 @@ iggy -u iggy -p iggy pat create my-token --store-token
iggy -n my-token stream list
```
-`--store-token` is mutually exclusive with an expiry: stored tokens **never expire**. Stored tokens are **namespaced per server address**, so a token stored for one server is not visible when connecting to another. Alternatively, create a token with an expiry and pass its value with `-t`:
+`--store-token` is mutually exclusive with an expiry: stored tokens **never expire**. They're also **namespaced per server address**, so a token stored for one server isn't visible when connecting to another. Alternatively, create a token with an expiry and pass its value with `-t`:
```bash
iggy -u iggy -p iggy pat create ci-token 7d
@@ -152,9 +152,9 @@ iggy context show production
iggy context delete production
```
-There is no `--context` flag: the active context is persistent state, switched with `iggy context use <name>`. Flags passed on the command line override the corresponding values from the active context.
+There's no `--context` flag: the active context is persistent state, switched with `iggy context use <name>`. Flags passed on the command line override the corresponding values from the active context.
-Contexts are stored in `contexts.toml`, and the active context name in `.active_context`, both under the Iggy home directory: `~/.iggy` by default, overridable with the `IGGY_HOME` environment variable. The `default` context always exists and **cannot be deleted or redefined**; deleting the currently active context switches back to `default`. Context names may contain letters, digits, hyphens and underscores.
+Contexts are stored in `contexts.toml`, and the active context name in `.active_context`, both under the Iggy home directory: `~/.iggy` by default, overridable with the `IGGY_HOME` environment variable. The `default` context always exists and **cannot be deleted or redefined**. Deleting the currently active context switches back to `default`. Context names may contain letters, digits, hyphens and underscores.
## Shell completions
diff --git a/content/docs/clustering/client-failover.mdx b/content/docs/clustering/client-failover.mdx
index 18995ef8..ebe9de48 100644
--- a/content/docs/clustering/client-failover.mdx
+++ b/content/docs/clustering/client-failover.mdx
@@ -3,7 +3,7 @@ title: Client Failover
description: Local reads, leader redirection, and retryable errors from the client side
---
-A clustered deployment changes what a client can expect from any single node: reads are served locally where possible, writes must reach the right primary, and the SDKs follow the primary as it moves. This page describes that behavior and which SDKs implement it.
+A clustered deployment changes what a client can expect from any single node: reads are served locally where possible, writes must reach the right primary, and the SDKs follow the primary as it moves.
## Where requests are served
@@ -18,9 +18,9 @@ One caveat on follower reads: server-managed offset auto-commit replicates the p
Leader-aware SDKs find the primary instead of requiring you to point them at it:
1. The client connects to any node from its connection string and fetches cluster metadata, which lists every node with its name, client endpoints, role, and status.
-2. When the connected node is not the leader, the client reconnects to the leader's advertised endpoint for the transport in use.
+2. When the connected node isn't the leader, the client reconnects to the leader's advertised endpoint for the transport in use.
3. When the cluster is transiently leaderless (for roughly one heartbeat timeout after a primary fails, while the election completes), the client polls the metadata until a leader appears instead of failing.
-4. Redirects are **capped** (three in the Rust SDK) so a flapping roster cannot bounce the client forever.
+4. Redirects are **capped** (three in the Rust SDK) so a flapping roster can't bounce the client forever.
The address a client is redirected to is the node's `advertised_address` (or the matching `advertised_addresses` selector), falling back to the roster `ip`. Getting those right matters: the client redials **exactly what the metadata advertises**. See [Configuration](/docs/clustering/configuration) for the selector rules.
diff --git a/content/docs/clustering/configuration.mdx b/content/docs/clustering/configuration.mdx
index c740ea0e..1bb5acb0 100644
--- a/content/docs/clustering/configuration.mdx
+++ b/content/docs/clustering/configuration.mdx
@@ -14,7 +14,7 @@ The server validates the whole section at boot and **refuses to start** on an in
| `enabled` | `false` | Enables cluster mode. When disabled, the node runs single-node and the rest of the section is mostly inert. |
| `name` | `"iggy-cluster"` | Unique cluster name shared by every node. Prevents accidental cross-cluster communication. |
-> `cluster.name` is a **permanent on-disk identity**, not a label. It is hashed into the cluster id stamped into every metadata and partition superblock on first boot, and the consensus layer derives its cluster id from the name unconditionally. Nodes with different names cannot connect to each other, and changing the name later makes the server refuse to start against existing data.
+> `cluster.name` is a **permanent on-disk identity**, not a label. It's hashed into the cluster id stamped into every metadata and partition superblock on first boot, and the consensus layer derives its cluster id from the name unconditionally. Nodes with different names cannot connect to each other, and changing the name later makes the server refuse to start against existing data.
## Consensus timing
@@ -38,7 +38,7 @@ Raise `heartbeat_timeout` on oversubscribed hosts where scheduling stalls fake p
Boot-time validation enforces these constraints:
-- Every duration must be **nonzero**. The `0`, `disabled`, and `unlimited` sentinels all parse to a zero duration and are rejected; there is no way to switch the liveness window off.
+- Every duration must be **nonzero**. The `0`, `disabled`, and `unlimited` sentinels all parse to a zero duration and are rejected. There's no way to switch the liveness window off.
- `heartbeat_timeout` must be at least `2s` and at least 4x `commit_broadcast_interval`, so one delayed broadcast never trips an election on a healthy primary.
- `view_change_status_timeout` must be at least 4x `view_change_retransmit_interval`, so a few dropped view-change messages retransmit instead of escalating into a fresh cluster-wide election.
- `view_probe_attempts_max` must be between 1 and 100.
@@ -63,9 +63,9 @@ Node name, unique within the roster and non-empty.
### `ip`
-The node's roster address. Replica-to-replica consensus traffic and follower-to-primary HTTP forwarding dial it, so it must be a **literal IPv4 or IPv6 address**; hostnames are rejected when the server resolves the roster at startup.
+The node's roster address. Replica-to-replica consensus traffic and follower-to-primary HTTP forwarding dial it, so it must be a **literal IPv4 or IPv6 address**. Hostnames are rejected when the server resolves the roster at startup.
-`ip` is **not the bind interface** for the client transports. `tcp`, `quic`, `http`, and `websocket` bind whatever their own `address` settings say; the roster supplies only their ports. The defaults bind loopback, so a cluster spread across hosts needs each transport's `address` set to `0.0.0.0` or the routable interface. The server warns at startup when a bind cannot serve the advertised `ip`.
+`ip` is **not the bind interface** for the client transports. `tcp`, `quic`, `http`, and `websocket` bind whatever their own `address` settings say. The roster supplies only their ports. The defaults bind loopback, so a cluster spread across hosts needs each transport's `address` set to `0.0.0.0` or the routable interface. The server warns at startup when a bind cannot serve the advertised `ip`.
### `replica_id`
@@ -96,7 +96,7 @@ client_cidr = "10.0.0.0/16" # in-VPC clients stay private
address = "10.0.1.5"
```
-Clients whose transport-level peer IP falls inside `client_cidr` are handed `address` instead of the catch-all. The **longest matching prefix** wins; clients matching no selector fall back to `advertised_address`, then `ip`. `address` takes the same forms as `advertised_address` (literal IP or hostname, never a port).
+Clients whose transport-level peer IP falls inside `client_cidr` are handed `address` instead of the catch-all. The **longest matching prefix** wins. Clients matching no selector fall back to `advertised_address`, then `ip`. `address` takes the same forms as `advertised_address` (literal IP or hostname, never a port).
Rules and caveats:
diff --git a/content/docs/clustering/deploy.mdx b/content/docs/clustering/deploy.mdx
index c5707872..507a2902 100644
--- a/content/docs/clustering/deploy.mdx
+++ b/content/docs/clustering/deploy.mdx
@@ -3,7 +3,7 @@ title: Deploy a Cluster
description: Build the binaries and run a three-node Apache Iggy cluster
---
-This page walks through building `iggy-server` and the `iggy` CLI, running a single development node, and bringing up a three-node cluster on one machine. See [Configuration](/docs/clustering/configuration) for the full `[cluster]` reference.
+Build `iggy-server` and the `iggy` CLI, run a single development node, then bring up a three-node cluster on one machine. See [Configuration](/docs/clustering/configuration) for the full `[cluster]` reference.
## Build from source
@@ -32,7 +32,7 @@ cargo run --bin iggy-server -- --with-default-root-credentials
`--with-default-root-credentials` sets the root user to `iggy`/`iggy` unless `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` are already exported. **Only the first boot** on an empty data directory reads these values.
-When clustering is disabled, `--replica-id 0` is still accepted; any other id is rejected because it would have to match a `cluster.nodes` entry.
+When clustering is disabled, `--replica-id 0` is still accepted. Any other id is rejected because it would have to match a `cluster.nodes` entry.
## Run a three-node cluster
@@ -79,7 +79,7 @@ export IGGY_ROOT_PASSWORD=iggy
export IGGY_CLUSTER_AUTH_SHARED_SECRET="replace-with-at-least-32-random-bytes"
```
-Root credentials are **mandatory** once the cluster is enabled. The shared secret must be **at least 32 bytes**; see [Security](/docs/clustering/security) for how it is used and rotated.
+Root credentials are **mandatory** once the cluster is enabled. The shared secret must be **at least 32 bytes**. See [Security](/docs/clustering/security) for how it's used and rotated.
Start each replica in a separate terminal. Use a different data path for every process:
@@ -102,12 +102,12 @@ In cluster mode:
- `replica_id` values must be unique and contiguous from `0`
- `ports` is the single source of listener ports: every enabled transport needs an explicit per-node port, otherwise the server **refuses to start**
- `tcp_replica` carries replica-to-replica consensus traffic and is **always required**
-- `ip` must be a **literal IP address**; use `advertised_address` when clients cannot reach it (see [Configuration](/docs/clustering/configuration))
+- `ip` must be a **literal IP address**. Use `advertised_address` when clients can't reach it (see [Configuration](/docs/clustering/configuration))
- use a different `system.path` for each process on the same host
## Spanning multiple hosts
-The roster `ip` is the address other replicas dial. It is **not the bind interface** for the client transports: `tcp`, `quic`, `http`, and `websocket` bind whatever their own `address` settings say, and the roster supplies only the port. The defaults listen on loopback, so a cluster spread across hosts needs each transport's `address` set to `0.0.0.0` or the routable interface. The server warns at startup when a bind cannot serve the advertised `ip`.
+The roster `ip` is the address other replicas dial. It's **not the bind interface** for the client transports: `tcp`, `quic`, `http`, and `websocket` bind whatever their own `address` settings say, and the roster supplies only the port. The defaults listen on loopback, so a cluster spread across hosts needs each transport's `address` set to `0.0.0.0` or the routable interface. The server warns at startup when a bind cannot serve the advertised `ip`.
## Containers
@@ -163,4 +163,4 @@ The cross-SDK BDD suites run against a real cluster started from `bdd/docker-com
./scripts/run-bdd-tests.sh rust leader_redirection
```
-The first argument selects the SDK (`rust`, `python`, `php`, `go`, `go-race`, `node`, `csharp`, `java`, `cpp`, or `all`) and the optional second argument selects one feature (`basic_messaging`, `leader_redirection`, `raw_command`, or `all`). `leader_redirection` is the clustered feature; it is supported by the Rust, Go, C#, and Java suites.
+The first argument selects the SDK (`rust`, `python`, `php`, `go`, `go-race`, `node`, `csharp`, `java`, `cpp`, or `all`) and the optional second argument selects one feature (`basic_messaging`, `leader_redirection`, `raw_command`, or `all`). `leader_redirection` is the clustered feature. The Rust, Go, C#, and Java suites support it.
diff --git a/content/docs/clustering/security.mdx b/content/docs/clustering/security.mdx
index 6673d438..7391a382 100644
--- a/content/docs/clustering/security.mdx
+++ b/content/docs/clustering/security.mdx
@@ -22,13 +22,13 @@ shared_secret = ""
## Rotating the shared secret
-`previous_shared_secret` enables a rolling key rotation without an authentication outage. The retiring key is accepted *for verification only*; every MAC a node produces uses `shared_secret`. Rotate in three rolls, restarting every node after each step:
+`previous_shared_secret` enables a rolling key rotation without an authentication outage. The retiring key is accepted *for verification only*. Every MAC a node produces uses `shared_secret`. Rotate in three rolls, restarting every node after each step:
1. `shared_secret = old`, `previous_shared_secret = new` on every node
2. `shared_secret = new`, `previous_shared_secret = old` on every node
3. `shared_secret = new` alone, closing the window
-Leave `previous_shared_secret` **empty outside a rotation**. It has the same 32-byte floor and the same environment-variable preference as the primary key (`IGGY_CLUSTER_AUTH_PREVIOUS_SHARED_SECRET`), and it must differ from `shared_secret`: an identical value is refused at boot because such a rotation would be a no-op.
+Leave `previous_shared_secret` **empty outside a rotation**. It has the same 32-byte floor and the same environment-variable preference as the primary key (`IGGY_CLUSTER_AUTH_PREVIOUS_SHARED_SECRET`), and it must differ from `shared_secret`. An identical value is refused at boot because the rotation would be a no-op.
## Replica TLS
@@ -46,9 +46,9 @@ key_file = "/etc/iggy/tls/node.key"
ca_file = "/etc/iggy/tls/ca.crt"
```
-TLS requires `cluster.auth.enabled = true`, and the server **refuses to start** otherwise. The reason: replica TLS carries no client certificates, so it authenticates the acceptor only. The PSK handshake authenticates the peer; TLS supplies confidentiality. Without the PSK requirement, any TLS-capable host could register as a replica.
+TLS requires `cluster.auth.enabled = true`, and the server **refuses to start** otherwise. The reason: replica TLS carries no client certificates, so it authenticates the acceptor only. The PSK handshake authenticates the peer. TLS supplies confidentiality. Without the PSK requirement, any TLS-capable host could register as a replica.
-Enabling TLS is also a coordinated-restart change: a TLS dialer cannot talk to a plaintext acceptor or vice versa.
+Enabling TLS is also a coordinated-restart change: a TLS dialer can't talk to a plaintext acceptor or vice versa.
Two certificate modes exist:
@@ -62,6 +62,6 @@ Follower nodes forward control-plane HTTP requests (stream, topic, user, and sim
- explicitly configured `http.jwt` signing secrets, identical on every node, or
- the cluster PSK: when HTTP is enabled, cluster auth is enabled, and no `http.jwt` secrets are configured, the signing key is derived from `shared_secret`.
-Either way, a bearer token minted on any node verifies on every node, which is the invariant the forwarding depends on. Forwarded requests dial the primary at its roster `ip` and `ports.http`; the follower verifies the caller's bearer locally before forwarding, and the primary re-authenticates the request through its normal stack.
+Either way, a bearer token minted on any node verifies on every node, which is the invariant the forwarding depends on. Forwarded requests dial the primary at its roster `ip` and `ports.http`. The follower verifies the caller's bearer locally before forwarding, and the primary re-authenticates the request through its normal stack.
The forwarding covers the control plane only. Partition-plane writes over HTTP (producing messages, storing consumer offsets) are **not forwarded**: each partition is its own consensus group whose primary can diverge from the metadata primary, so those requests must land on the right node. See [Client failover](/docs/clustering/client-failover) for how clients find it.
diff --git a/content/docs/clustering/vsr.mdx b/content/docs/clustering/vsr.mdx
index 222ddc59..bfc89051 100644
--- a/content/docs/clustering/vsr.mdx
+++ b/content/docs/clustering/vsr.mdx
@@ -5,11 +5,11 @@ description: How Apache Iggy replicates data across a cluster with VSR
Apache Iggy replicates clusters with **Viewstamped Replication Revisited (VSR)**. VSR keeps an ordered state machine consistent across replicas and elects a new primary when the current primary fails.
-Clustering is built into the standard `iggy-server` binary. There is **no separate build, binary, or Cargo feature**: the same server runs single-node or clustered, and `cluster.enabled` in the configuration decides which one you get. Clustering is **disabled by default**.
+Clustering is built into the standard `iggy-server` binary. There's **no separate build, binary, or Cargo feature**: the same server runs single-node or clustered, and `cluster.enabled` in the configuration decides which one you get. Clustering is **disabled by default**.
Read the [VSR paper used by the project](https://github.com/apache/iggy/blob/master/assets/vsr.pdf) for the protocol specification.
-> Iggy has not reached a 1.0 production release yet. The cluster protocol and configuration described here can still change between pre-release versions.
+> Iggy hasn't reached a 1.0 production release yet. The cluster protocol and configuration described here can still change between pre-release versions.
## Current implementation
@@ -95,7 +95,7 @@ async fn main() -> Result<(), IggyError> {
## Status
-VSR started as a parallel `server-ng` implementation behind a `vsr` build feature. That phase is over: the VSR server was promoted to the mainline `core/server` crate and the `iggy-server` binary, the feature flag was removed, and the legacy server together with its wire format was deleted. What this page describes is the only server implementation.
+VSR started as a parallel `server-ng` implementation behind a `vsr` build feature. That phase is over. The VSR server moved into the mainline `core/server` crate and the `iggy-server` binary, the feature flag is gone, and the legacy server and its wire format were deleted. What this page describes is the only server implementation.
## Next steps
diff --git a/content/docs/connectors/introduction.mdx b/content/docs/connectors/introduction.mdx
index 59ef8815..7a65dc90 100644
--- a/content/docs/connectors/introduction.mdx
+++ b/content/docs/connectors/introduction.mdx
@@ -62,7 +62,7 @@ Messages can be decoded and encoded using the following formats: **JSON**, **Raw
## Quick Start
-1. Build the project in release mode (or debug, and update the connectors paths in the config accordingly), and make sure that the plugins specified in `core/connectors/runtime/example_config/connectors/` directory under `path` are available. The configuration must be provided in `toml` format. Note that the example directory enables connectors for ClickHouse, Delta Lake, Apache Doris, Apache Iceberg, and InfluxDB as well - without their backing services (or their compiled plugins) these [...]
+1. Build the project in release mode (or debug, and update the connectors paths in the config accordingly), and make sure that the plugins specified in `core/connectors/runtime/example_config/connectors/` directory under `path` are available. The configuration must be provided in `toml` format. The example directory also enables connectors for ClickHouse, Delta Lake, Apache Doris, Apache Iceberg, and InfluxDB. Without their backing services (or their compiled plugins) these are reported [...]
2. Run `docker compose up -d` from `/examples/rust/src/sink-data-producer` which will start the Quickwit server to be used by an example sink connector. At this point, you can access the Quickwit UI at [http://localhost:7280](http://localhost:7280) - check this dashboard again later on, after the `events` index will be created.
@@ -83,7 +83,7 @@ Messages can be decoded and encoded using the following formats: **JSON**, **Raw
## Configuration
-The same rules apply when it comes to overriding the configuration via environment variables as for the main [Iggy server](/docs/server/configuration).
+Overriding the configuration via environment variables follows the same rules as the main [Iggy server](/docs/server/configuration).
You can provide the following envs during the runtime startup:
@@ -110,7 +110,7 @@ Each sink should have its own, custom configuration, which is passed along with
## Source
-Sources are responsible for producing the messages to the configured stream(s) and topic(s). For example, the Random source connector will generate the random messages that will be then sent to the configured stream and topic.
+Sources are responsible for producing the messages to the configured stream(s) and topic(s). For example, the Random source connector generates random messages that are then sent to the configured stream and topic.
Please refer to the **[Source documentation](/docs/connectors/sources/source)** for the details about the configuration and the sample implementation.
diff --git a/content/docs/connectors/observability.mdx b/content/docs/connectors/observability.mdx
index 2de94cf2..0ca8b65b 100644
--- a/content/docs/connectors/observability.mdx
+++ b/content/docs/connectors/observability.mdx
@@ -45,14 +45,12 @@ Runtime-level gauges:
Per-connector counters, labeled with `connector_key` and `connector_type` (`source` or `sink`):
-- `iggy_connector_messages_produced_total_total` - messages received from a source plugin's poll.
-- `iggy_connector_messages_sent_total_total` - messages sent to Iggy (source).
-- `iggy_connector_messages_consumed_total_total` - messages consumed from Iggy (sink).
-- `iggy_connector_messages_processed_total_total` - messages processed and delivered to the sink plugin.
-- `iggy_connector_messages_filtered_total_total` - messages intentionally dropped by transforms.
-- `iggy_connector_errors_total_total` - errors encountered.
-
-The doubled `_total_total` suffix is what a scraper sees today: the counters are registered under names that already end in `_total`, and the Prometheus client library appends its own `_total` suffix to every counter at encode time. The names may normalize to a single suffix in a future release, so check `/metrics` after upgrading.
+- `iggy_connector_messages_produced_total` - messages received from a source plugin's poll.
+- `iggy_connector_messages_sent_total` - messages sent to Iggy (source).
+- `iggy_connector_messages_consumed_total` - messages consumed from Iggy (sink).
+- `iggy_connector_messages_processed_total` - messages processed and delivered to the sink plugin.
+- `iggy_connector_messages_filtered_total` - messages intentionally dropped by transforms.
+- `iggy_connector_errors_total` - errors encountered.
Per-batch stage timings are recorded in the `iggy_connector_stage_duration_seconds` histogram, labeled with `connector_key`, `connector_type`, and `stage` (`prepare`, `ffi`, `decode`, `iggy_send`, `state_save`, `total`), with buckets ranging from 50 microseconds to 5 seconds.
@@ -89,6 +87,6 @@ Source connectors can persist their position (see the [SDK documentation](/docs/
path = "local_state"
```
-Each source gets one file named `source_{key}.state` (e.g. `local_state/source_random.state`) holding the raw state bytes - by convention MessagePack produced by the SDK helpers. Sinks have no state files; their progress lives in Iggy consumer groups.
+Each source gets one file named `source_{key}.state` (e.g. `local_state/source_random.state`) holding the raw state bytes - by convention MessagePack produced by the SDK helpers. Sinks have no state files. Their progress lives in Iggy consumer groups.
-The write is crash-safe: the state is written and fsynced to a `.tmp` file, atomically renamed over the previous file, and the directory entry is fsynced. Files are created with owner-only permissions (`0600`), since state may carry cursors or tokens. An empty or missing state file simply means the source starts fresh - deleting a state file resets that source's position.
+The write is crash-safe: the state is written and fsynced to a `.tmp` file, atomically renamed over the previous file, and the directory entry is fsynced. Files are created with owner-only permissions (`0600`), since state may carry cursors or tokens. An empty or missing state file means the source starts fresh - deleting a state file resets that source's position.
diff --git a/content/docs/connectors/runtime.mdx b/content/docs/connectors/runtime.mdx
index 9b923703..16a31f6b 100644
--- a/content/docs/connectors/runtime.mdx
+++ b/content/docs/connectors/runtime.mdx
@@ -65,7 +65,7 @@ The runtime supports two types of configuration providers for managing connector
### Local File Provider
-The default configuration provider reads connector configurations from local files. Each connector (source or sink) is configured in its own separate file within the directory specified by `connectors.config_dir`. An empty `config_dir` is a fatal startup error; a missing directory is created automatically with a warning, and no connectors are loaded from it.
+The default configuration provider reads connector configurations from local files. Each connector (source or sink) is configured in its own separate file within the directory specified by `connectors.config_dir`. An empty `config_dir` is a fatal startup error. A missing directory is created automatically with a warning, and no connectors are loaded from it.
```toml
[connectors]
diff --git a/content/docs/connectors/sdk.mdx b/content/docs/connectors/sdk.mdx
index e78448be..e7a64ddd 100644
--- a/content/docs/connectors/sdk.mdx
+++ b/content/docs/connectors/sdk.mdx
@@ -8,7 +8,7 @@ Moreover, it contains both, the `decoders` and `encoders` modules, implementing
## Traits
-A source produces messages to the configured stream and topic, a sink consumes messages from the configured stream(s) and topic(s):
+A source produces messages to the configured stream and topic. A sink consumes messages from the configured stream(s) and topic(s):
```rust
#[async_trait]
@@ -53,7 +53,7 @@ The macros generate the C FFI symbols loaded by the runtime: `iggy_sink_open`, `
## State
-Source connectors can persist their position between restarts. Each batch of `ProducedMessages` may carry an optional `ConnectorState` - the runtime saves it to a file after the batch is sent to Iggy, and passes the last saved state to `new()` on the next startup. `ConnectorState` wraps raw bytes and provides `serialize`/`deserialize` helpers backed by MessagePack. Sinks do not manage state - their offsets are tracked by Iggy consumer groups. See the **[Observability page](/docs/connecto [...]
+Source connectors can persist their position between restarts. Each batch of `ProducedMessages` may carry an optional `ConnectorState` - the runtime saves it to a file after the batch is sent to Iggy, and passes the last saved state to `new()` on the next startup. `ConnectorState` wraps raw bytes and provides `serialize`/`deserialize` helpers backed by MessagePack. Sinks don't manage state. Their offsets are tracked by Iggy consumer groups. See the **[Observability page](/docs/connectors [...]
## Retry utilities
@@ -80,7 +80,7 @@ The SDK includes support for Protocol Buffers (protobuf) format with both encodi
### Configuration Example
-Here's a complete example configuration for using Protocol Buffers with Iggy connectors. Note that no protobuf-specific plugin ships with the runtime - the example assumes a custom source and sink you implement yourself; the `schema = "proto"` stream setting and the `proto_convert` transform work with any connector.
+Here's a complete example configuration for using Protocol Buffers with Iggy connectors. No protobuf-specific plugin ships with the runtime, so the example assumes a custom source and sink you implement yourself. The `schema = "proto"` stream setting and the `proto_convert` transform work with any connector.
**Main runtime config (config.toml):**
@@ -149,9 +149,9 @@ type_url_prefix = "type.googleapis.com"
strict_mode = false
```
-Note that the format-conversion transforms define no per-key defaults - every non-optional key shown above must be present, or the configuration fails to deserialize (`schema_path`, `message_type`, `field_mappings`, and `descriptor_set` are the optional ones).
+The format-conversion transforms define no per-key defaults. Every non-optional key shown above must be present, or the configuration fails to deserialize (`schema_path`, `message_type`, `field_mappings`, and `descriptor_set` are the optional ones).
-Note the differences between the two `[[streams]]` shapes: a source produces to a single `topic` and can tune batching via `batch_length` and `linger_time`, while a sink consumes from a list of `topics` and can additionally set `batch_length`, `poll_interval`, and `consumer_group`.
+The two `[[streams]]` shapes differ: a source produces to a single `topic` and can tune batching via `batch_length` and `linger_time`, while a sink consumes from a list of `topics` and can additionally set `batch_length`, `poll_interval`, and `consumer_group`.
### Key Configuration Options
@@ -170,7 +170,7 @@ Note the differences between the two `[[streams]]` shapes: a source produces to
- **`field_mappings`**: Mapping of field names for transformation (e.g., `"old_field" = "new_field"`)
- **`conversion_options`**: Fine-tuning knobs: `validate_messages`, `pretty_json`, `include_metadata`, `type_url_prefix`, `strict_mode`
-The `schema_registry_url` field is reserved and currently not implemented - the SDK never contacts a schema registry; schemas are loaded only from `schema_path` or `descriptor_set`.
+The `schema_registry_url` field is reserved and currently not implemented. The SDK never contacts a schema registry, and schemas are loaded only from `schema_path` or `descriptor_set`.
### Supported Features
diff --git a/content/docs/connectors/sinks/clickhouse.mdx b/content/docs/connectors/sinks/clickhouse.mdx
index d3cc2634..618a08c0 100644
--- a/content/docs/connectors/sinks/clickhouse.mdx
+++ b/content/docs/connectors/sinks/clickhouse.mdx
@@ -57,16 +57,16 @@ verbose_logging = false
## Insert Formats
- **`json_each_row`** (default): accepts JSON payloads. Each message is sent as one JSON object per line using ClickHouse's `JSONEachRow` format. ClickHouse coerces JSON values to the column types, so the table can have any schema.
-- **`row_binary`**: accepts JSON payloads and serializes them to ClickHouse's `RowBinaryWithDefaults` binary format, which is more efficient than JSON at high volume. The table must already exist; its schema is fetched from `system.columns` and validated at startup. Requires ClickHouse 23.7 or newer. Columns with a `DEFAULT` expression may be omitted from messages; `MATERIALIZED`, `ALIAS`, and `EPHEMERAL` columns are dropped from the schema. Unsupported column types (128/256-bit integers [...]
+- **`row_binary`**: accepts JSON payloads and serializes them to ClickHouse's `RowBinaryWithDefaults` binary format, which is more efficient than JSON at high volume. The table must already exist. Its schema is fetched from `system.columns` and validated at startup. Requires ClickHouse 23.7 or newer. Columns with a `DEFAULT` expression may be omitted from messages. `MATERIALIZED`, `ALIAS`, and `EPHEMERAL` columns are dropped from the schema. Unsupported column types (128/256-bit integers [...]
- **`string`**: accepts text payloads and passes them through unchanged. Use `string_format` to tell ClickHouse which format the payload is in (`csv`, `tsv`, or `json_each_row`).
-In `row_binary` mode the schema is captured once at startup and the insert stream is positional. An `ALTER TABLE` on the target while the connector runs can silently corrupt inserted rows, so restart the connector after any schema change. The self-describing `json_each_row` format (including `string` mode with `string_format = "json_each_row"`) maps values by field name and tolerates schema changes; `string` passthrough with `csv` or `tsv` sends plain positional ClickHouse formats mapped [...]
+In `row_binary` mode the schema is captured once at startup and the insert stream is positional. An `ALTER TABLE` on the target while the connector runs can silently corrupt inserted rows, so restart the connector after any schema change. The self-describing `json_each_row` format (including `string` mode with `string_format = "json_each_row"`) maps values by field name and tolerates schema changes. `string` passthrough with `csv` or `tsv` sends plain positional ClickHouse formats mapped [...]
## Error Handling & Delivery Semantics
-Failed inserts are attempted up to `max_retries` total times with exponential backoff and full jitter, starting from `retry_delay`. A message whose payload type does not match the chosen format is skipped with an error log and the rest of the batch is still sent, in every mode. In `row_binary` mode there is an additional failure class: a JSON row whose values cannot be converted to the column types fails the whole batch (a half-written binary row would corrupt the rows after it).
+Failed inserts are attempted up to `max_retries` total times with exponential backoff and full jitter, starting from `retry_delay`. A message whose payload type doesn't match the chosen format is skipped with an error log and the rest of the batch is still sent, in every mode. In `row_binary` mode there is an additional failure class: a JSON row whose values cannot be converted to the column types fails the whole batch (a half-written binary row would corrupt the rows after it).
-Delivery is **at-least-once**: retries resend the full batch without an `insert_deduplication_token`, so a lost acknowledgement can insert the same rows twice. `ReplicatedMergeTree` tables suppress duplicates via implicit block-level deduplication in the common retry case; plain `MergeTree` tables store them. If duplicates matter, deduplicate at the table engine or query level as described in the upstream [clickhouse_sink README](https://github.com/apache/iggy/tree/master/core/connectors [...]
+Delivery is **at-least-once**: retries resend the full batch without an `insert_deduplication_token`, so a lost acknowledgement can insert the same rows twice. `ReplicatedMergeTree` tables suppress duplicates via implicit block-level deduplication in the common retry case. Plain `MergeTree` tables store them. If duplicates matter, deduplicate at the table engine or query level as described in the upstream [clickhouse_sink README](https://github.com/apache/iggy/tree/master/core/connectors [...]
## Transforms
diff --git a/content/docs/connectors/sinks/delta.mdx b/content/docs/connectors/sinks/delta.mdx
index f8e29b41..a0c569c8 100644
--- a/content/docs/connectors/sinks/delta.mdx
+++ b/content/docs/connectors/sinks/delta.mdx
@@ -29,7 +29,7 @@ consumer_group = "delta_sink_connector"
table_uri = "file:///tmp/iggy_delta_table"
```
-The target Delta table must already exist at `table_uri`: the connector opens it at startup and fails if it is missing. It does not create tables. Only JSON payloads are supported (`schema = "json"` on the stream); a batch containing another payload type fails.
+The target Delta table must already exist at `table_uri`: the connector opens it at startup and fails if it's missing. It doesn't create tables. Only JSON payloads are supported (`schema = "json"` on the stream). A batch containing another payload type fails.
### Options
@@ -42,7 +42,7 @@ Each backend has its own credential options, required when `storage_backend_type
- **S3**: `aws_s3_access_key`, `aws_s3_secret_key`, `aws_s3_region`, `aws_s3_endpoint_url` (for S3-compatible stores like MinIO), `aws_s3_allow_http` (default `false`, for local development).
- **Azure**: `azure_storage_account_name`, `azure_container_name`, and either `azure_storage_account_key` or `azure_storage_sas_token` (not both).
-- **GCS**: `gcs_service_account_key` (the service account JSON as a string; the bucket comes from the `gs://` URI).
+- **GCS**: `gcs_service_account_key` (the service account JSON as a string, with the bucket taken from the `gs://` URI).
### S3 example
@@ -59,15 +59,15 @@ aws_s3_region = "us-east-1"
JSON values are coerced to the Delta table schema before writing:
-- **Timestamp columns**: ISO 8601 / RFC 3339 strings (e.g. `"2021-11-11T22:11:58Z"`) are converted to microsecond timestamps; numeric timestamps pass through.
+- **Timestamp columns**: ISO 8601 / RFC 3339 strings (e.g. `"2021-11-11T22:11:58Z"`) are converted to microsecond timestamps. Numeric timestamps pass through.
- **String columns**: non-string values are converted to their string representation.
- Coercions apply recursively to nested structs and arrays.
-The table schema is captured once at startup and not refreshed, so restart the connector after changing the table schema.
+The table schema is captured once at startup and not refreshed, so restart the connector after changing it.
## Batching & Transactions
-Every poll batch is written to Parquet buffers and committed to the Delta log as a single transaction. The stream's `batch_length` therefore controls Delta commit granularity: small batches produce many small files and table versions, so prefer larger batches for production tables (and compact periodically). There is no retry on failed writes; a failed batch surfaces as a connector error.
+Every poll batch is written to Parquet buffers and committed to the Delta log as a single transaction. The stream's `batch_length` therefore controls Delta commit granularity: small batches produce many small files and table versions, so prefer larger batches for production tables (and compact periodically). There is no retry on failed writes. A failed batch surfaces as a connector error.
## Transforms
diff --git a/content/docs/connectors/sinks/doris.mdx b/content/docs/connectors/sinks/doris.mdx
index e465336b..9ed12825 100644
--- a/content/docs/connectors/sinks/doris.mdx
+++ b/content/docs/connectors/sinks/doris.mdx
@@ -36,7 +36,7 @@ batch_size = 1000
timeout = "30s"
```
-The target database and table must be pre-created; the connector never issues DDL. `database` and `table` must match `[A-Za-z0-9_]+` or startup fails. Streams must use `schema = "json"`.
+The target database and table must be pre-created. The connector never issues DDL. `database` and `table` must match `[A-Za-z0-9_]+` or startup fails. Streams must use `schema = "json"`.
### Common Options
@@ -58,19 +58,19 @@ Further options cover the label prefix, connect timeout, retry backoff (`retry_d
## Output Formats
- **`json`** (default): payloads are sent as a JSON array with `strip_outer_array: true`. Values map to columns by name, so the JSON shape must match the table (use `columns` if the JSON field names differ from the table columns or you need derived expressions).
-- **`csv`**: opt-in for throughput. Doris CSV is positional, so `output_format = "csv"` **requires `columns`** to pin the column order; startup fails without it. Rows are framed with control-character separators and quoted with `enclose`/`escape`, so embedded commas and newlines are safe. JSON `null` and missing keys become SQL `NULL`, and nested objects or arrays are stringified as JSON.
+- **`csv`**: opt-in for throughput. Doris CSV is positional, so `output_format = "csv"` **requires `columns`** to pin the column order. Startup fails without it. Rows are framed with control-character separators and quoted with `enclose`/`escape`, so embedded commas and newlines are safe. JSON `null` and missing keys become SQL `NULL`, and nested objects or arrays are stringified as JSON.
## Delivery Semantics
Each batch is loaded under a deterministic label derived from the stream, topic, partition, and offset range. Transient failures (HTTP 5xx/408/429, transport errors) are retried in-request with exponential backoff and jitter under the same label, which Doris deduplicates, so an ambiguous success is absorbed rather than doubled. Permanent errors (4xx, `Fail` status, malformed responses) are never retried.
-The runtime commits consumer offsets at poll time and does not inspect `consume()`'s return value, so delivery is **at-most-once** across polls: a failure that outlives the retry budget, or a crash mid-load, is not replayed. Keep `batch_size` stable across a manual redrive, since changing it shifts chunk boundaries and defeats label deduplication.
+The runtime commits consumer offsets at poll time and doesn't inspect `consume()`'s return value, so delivery is **at-most-once** across polls: a failure that outlives the retry budget, or a crash mid-load, is not replayed. Keep `batch_size` stable across a manual redrive, since changing it shifts chunk boundaries and defeats label deduplication.
## Security Notes
- Use `https://` for `fe_url` in production: credentials travel as HTTP Basic auth.
- The connector preserves the `Authorization` header across the Doris FE-to-BE 307 redirect, validating the target first: scheme downgrades are refused unless `allow_insecure_redirect = true`, and `allowed_redirect_hosts` can pin redirects to known backend endpoints.
-- `columns` and `where` are forwarded verbatim to Doris and evaluated as SQL expressions; keep this config trusted.
+- `columns` and `where` are forwarded verbatim to Doris and evaluated as SQL expressions. Keep this config trusted.
## Transforms
diff --git a/content/docs/connectors/sinks/elasticsearch.mdx b/content/docs/connectors/sinks/elasticsearch.mdx
index fa659181..cc357cdb 100644
--- a/content/docs/connectors/sinks/elasticsearch.mdx
+++ b/content/docs/connectors/sinks/elasticsearch.mdx
@@ -42,7 +42,7 @@ create_index_if_not_exists = true
| `batch_size` | integer | unused | Accepted but currently not read by the connector |
| `timeout_seconds` | integer | unused | Accepted but currently not read by the connector |
-Batching is governed by the stream-level `batch_length`: each consume cycle bulk-indexes the batch of messages it receives. The connector performs no retries of its own; failed batches are logged and counted.
+The stream-level `batch_length` governs batching: each consume cycle bulk-indexes the batch of messages it receives. The connector performs no retries of its own. Failed batches are logged and counted.
### Index mapping
@@ -65,7 +65,7 @@ type = "keyword"
## Payload handling
- JSON payloads are indexed as documents directly.
-- Raw payloads are parsed as JSON when possible; otherwise they are wrapped as `{ "data": "<base64>", "data_type": "raw" }`.
+- Raw payloads are parsed as JSON when possible, and wrapped as `{ "data": "<base64>", "data_type": "raw" }` when that fails.
- Text payloads are wrapped as `{ "text": "...", "data_type": "text" }`.
- Other payload formats are skipped with a warning.
diff --git a/content/docs/connectors/sinks/http.mdx b/content/docs/connectors/sinks/http.mdx
index 7dae535b..846fffb1 100644
--- a/content/docs/connectors/sinks/http.mdx
+++ b/content/docs/connectors/sinks/http.mdx
@@ -2,7 +2,7 @@
title: HTTP Sink
---
-The HTTP Sink connector consumes messages from Iggy streams and delivers them to any HTTP endpoint: webhooks, REST APIs, serverless functions, or SaaS integrations. This is the transport-level, generic HTTP connector. It is distinct from sinks such as Elasticsearch or Quickwit, which use HTTP internally to talk to one specific system. You bring the URL, headers, and batching strategy, and the sink handles transport, retries, and metadata wrapping.
+The HTTP Sink connector consumes messages from Iggy streams and delivers them to any HTTP endpoint: webhooks, REST APIs, serverless functions, or SaaS integrations. This is the transport-level, generic HTTP connector. It's distinct from sinks such as Elasticsearch or Quickwit, which use HTTP internally to talk to one specific system. You bring the URL, headers, and batching strategy, and the sink handles transport, retries, and metadata wrapping.
This page is a curated subset of the documentation. The canonical reference, including the full configuration table, deployment patterns, and performance analysis, is the upstream [http_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/http_sink) in the `apache/iggy` repository.
@@ -46,7 +46,7 @@ The `batch_mode` option controls how messages from one poll cycle are delivered
- **`json_array`**: all messages as a single JSON array. Best for APIs that expect array payloads. `Content-Type: application/json`.
- **`raw`**: raw bytes, one request per message. For non-JSON payloads (Protobuf, FlatBuffers, binary). The metadata envelope is not applied. `Content-Type: application/octet-stream`.
-Both `individual` and `raw` perform one HTTP request per message. For production throughput with JSON-representable payloads, prefer `ndjson` or `json_array`, which collapse N round trips per poll cycle into one.
+For production throughput with JSON-representable payloads, prefer `ndjson` or `json_array`, which collapse N round trips per poll cycle into one.
## Metadata Envelope
@@ -80,7 +80,7 @@ Non-JSON payloads (`raw`, `flatbuffer`, `proto`, and `avro` schemas) cannot be e
With the envelope enabled, this object becomes the `payload` field shown above.
-The connector does not require any particular message structure on input. The envelope is applied on the way out, not expected on the way in, so your producers can publish whatever they like.
+The connector doesn't require any particular message structure on input. The envelope is applied on the way out, not expected on the way in, so your producers can publish whatever they like.
## Authentication
@@ -99,11 +99,11 @@ Failed requests are retried with exponential backoff: `retry_delay`, multiplied
- **Transient errors** (retried): network errors, HTTP 429, 500, 502, 503, 504.
- **Non-transient errors** (fail immediately): HTTP 400, 401, 403, 404, 405, etc.
-- **`success_status_codes`** (default `[200, 201, 202, 204]`) short-circuits retries: any status code in this set is treated as success and is never retried, even codes that are normally transient. Placing `429` in the set makes the sink accept rate-limited responses as delivered; the sink warns at startup about such overlaps.
-- **HTTP 429 `Retry-After`**: the header is logged but not honored; retry timing always uses the computed backoff.
+- **`success_status_codes`** (default `[200, 201, 202, 204]`) short-circuits retries: any status code in this set is treated as success and is never retried, even codes that are normally transient. Placing `429` in the set makes the sink accept rate-limited responses as delivered. The sink warns at startup about such overlaps.
+- **HTTP 429 `Retry-After`**: the header is logged but not honored. Retry timing always uses the computed backoff.
- **Partial delivery** (`individual`/`raw` modes): after a fixed number of consecutive HTTP failures (3 at the time of writing, not configurable), the remainder of the batch is aborted to avoid hammering a dead endpoint.
-The connector runtime commits consumer-group offsets when messages are polled, before `consume()` runs, and does not inspect its return value, so the effective delivery guarantee is **at-most-once** at the runtime level. The sink's internal retry loop provides best-effort delivery within each `consume()` call.
+The connector runtime commits consumer-group offsets when messages are polled, before `consume()` runs, and doesn't inspect its return value, so the effective delivery guarantee is **at-most-once** at the runtime level. The sink's internal retry loop provides best-effort delivery within each `consume()` call.
## Example Configurations
@@ -117,7 +117,7 @@ include_metadata = false
max_retries = 5
```
-The sink performs no outbound payload transformation. `include_metadata = false` only skips the envelope; the message payload is delivered exactly as produced, so your producer must publish JSON already in the shape Slack expects (for example `{"text": "..."}`).
+The sink performs no outbound payload transformation. `include_metadata = false` only skips the envelope. The message payload is delivered exactly as produced, so your producer must publish JSON already in the shape Slack expects (for example `{"text": "..."}`).
### Bulk REST API Ingestion
@@ -153,9 +153,9 @@ timeout = "5s"
## Deployment & Performance
-One `iggy-connectors` runtime process can host many connectors: the local config provider imports every connector TOML file under its `config_dir`, and each entry gets its own plugin instance and consume tasks in the same process. Each connector entry has exactly one `[plugin_config]` block and therefore one destination URL, so to send to multiple destinations, add multiple HTTP sink entries with distinct keys and URLs (in the same runtime or separate ones). For fan-out from a single top [...]
+One `iggy-connectors` runtime process can host many connectors: the local config provider imports every connector TOML file under its `config_dir`, and each entry gets its own plugin instance and consume tasks in the same process. Each connector entry has exactly one `[plugin_config]` block and therefore one destination URL. To send to multiple destinations, add multiple HTTP sink entries with distinct keys and URLs (in the same runtime or separate ones). For fan-out from a single topic [...]
-Throughput is dominated by batch-mode choice: the runtime calls `consume()` sequentially within each topic task, so `individual` and `raw` modes perform N round trips per poll cycle while `ndjson`/`json_array` collapse the same batch into one request. Connection pooling and HTTP keep-alive are enabled by default; tune `max_connections` and `batch_length` together for high-throughput targets.
+Throughput is dominated by batch-mode choice: the runtime calls `consume()` sequentially within each topic task, so `individual` and `raw` modes perform N round trips per poll cycle while `ndjson`/`json_array` collapse the same batch into one request. Connection pooling and HTTP keep-alive are enabled by default. Tune `max_connections` and `batch_length` together for high-throughput targets.
For multi-instance deployment patterns (Docker, Kubernetes, fan-out topologies), connection pool tuning, and full performance analysis, see the upstream [http_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/http_sink).
@@ -163,7 +163,7 @@ For multi-instance deployment patterns (Docker, Kubernetes, fan-out topologies),
- **No `Retry-After` honoring**: the header is logged but doesn't influence retry timing.
- **No dead-letter queue**: failed messages are logged at `error!` level but not persisted.
-- **No request signing**: AWS SigV4, HMAC, and similar schemes are not supported. Use custom headers or a signing proxy.
+- **No request signing**: AWS SigV4, HMAC, and similar schemes aren't supported. Use custom headers or a signing proxy.
- **No per-topic URL routing**: all topics in one connector instance share the same `url`. For routing, deploy multiple instances.
- **No OAuth2 / OIDC token refresh**: bearer tokens are static. Use an auth proxy for services that require token rotation.
- **No mTLS client certificates**: terminate mTLS at a sidecar proxy for production use.
diff --git a/content/docs/connectors/sinks/iceberg.mdx b/content/docs/connectors/sinks/iceberg.mdx
index 72dec98a..f960458f 100644
--- a/content/docs/connectors/sinks/iceberg.mdx
+++ b/content/docs/connectors/sinks/iceberg.mdx
@@ -40,7 +40,7 @@ store_class = "s3"
- **store_url**: The URL of the object storage for data uploads.
- **store_access_key_id**: The access key ID of the object storage.
- **store_secret_access_key**: The secret key used to upload data to the object storage.
-- **store_region**: The region of the object storage. Required; for S3-compatible stores that ignore it, supply any placeholder value.
+- **store_region**: The region of the object storage. Required. For S3-compatible stores that ignore it, supply any placeholder value.
- **store_class**: The storage class to use. **Currently, only S3-compatible storage is supported.**
## Dynamic Routing
@@ -97,4 +97,4 @@ enabled = true
field = "data"
```
-Set `field` to the envelope key that contains the actual row data. See the [transforms documentation](/docs/connectors/transforms) and your source connector's documentation for details on the envelope shape.
+Set `field` to the envelope key that holds the row data. See the [transforms documentation](/docs/connectors/transforms) and your source connector's documentation for details on the envelope shape.
diff --git a/content/docs/connectors/sinks/influxdb.mdx b/content/docs/connectors/sinks/influxdb.mdx
index 10255b5f..b90e0acd 100644
--- a/content/docs/connectors/sinks/influxdb.mdx
+++ b/content/docs/connectors/sinks/influxdb.mdx
@@ -36,7 +36,7 @@ precision = "us"
batch_size = 500
```
-InfluxDB V2 organizes data as `org` + `bucket`; V3 uses a single `db` field instead. Omitting `version` defaults to `"v2"` for backward compatibility. The write body, batching, and retry behavior are identical between versions; the connector handles the differing endpoints, auth header styles, and precision spellings internally.
+InfluxDB V2 organizes data as `org` + `bucket`. V3 uses a single `db` field instead. Omitting `version` defaults to `"v2"` for backward compatibility. The write body, batching, and retry behavior are identical between versions. The connector handles the differing endpoints, auth header styles, and precision spellings internally.
```toml
# V3 variant: org/bucket are replaced by db
@@ -63,7 +63,7 @@ token = "replace_with_secret_token"
| `payload_format` | string | `json` | `json`, `text`, or `base64` |
| `include_metadata` | bool | `true` | Inject stream/topic/partition fields into each point (the `offset` tag is always written regardless) |
-The `include_checksum`, `include_origin_timestamp`, `include_stream_tag`, `include_topic_tag`, and `include_partition_tag` flags also default to `true`; disable them individually to slim down points. Resilience options cover `timeout` (`30s`), `max_retries` (`3`), `retry_delay` / `retry_max_delay`, startup health-check retries, and a circuit breaker (`circuit_breaker_threshold` `5`, `circuit_breaker_cool_down` `30s`). See the upstream [influxdb_sink README](https://github.com/apache/iggy [...]
+The `include_checksum`, `include_origin_timestamp`, `include_stream_tag`, `include_topic_tag`, and `include_partition_tag` flags also default to `true`. Disable them individually to slim down points. Resilience options cover `timeout` (`30s`), `max_retries` (`3`), `retry_delay` / `retry_max_delay`, startup health-check retries, and a circuit breaker (`circuit_breaker_threshold` `5`, `circuit_breaker_cool_down` `30s`). See the upstream [influxdb_sink README](https://github.com/apache/iggy [...]
## Payload Formats
diff --git a/content/docs/connectors/sinks/meilisearch.mdx b/content/docs/connectors/sinks/meilisearch.mdx
index 1335748a..a9f3aaaa 100644
--- a/content/docs/connectors/sinks/meilisearch.mdx
+++ b/content/docs/connectors/sinks/meilisearch.mdx
@@ -53,15 +53,15 @@ Further options cover task polling (`task_timeout` `30s`, `task_poll_interval` `
## Document Mapping
-JSON object payloads are indexed as documents directly. JSON arrays and scalars are wrapped in a `value` field, since Meilisearch documents must be objects. Raw payloads are parsed as JSON when possible and otherwise indexed as base64 data; text payloads land in a `text` field. Records with unsupported payload schemas are skipped with a warning and counted as errors.
+JSON object payloads are indexed as documents directly. JSON arrays and scalars are wrapped in a `value` field, since Meilisearch documents must be objects. Raw payloads are parsed as JSON when possible and otherwise indexed as base64 data. Text payloads land in a `text` field. Records with unsupported payload schemas are skipped with a warning and counted as errors.
-When the configured primary key is absent from a document, the connector injects a stable value derived from the stream, topic, partition, offset, and message ID, avoiding Meilisearch primary-key inference failures. If your payloads carry their own primary key, its values must be unique, otherwise add-or-replace semantics collapse distinct messages into one document.
+When the configured primary key is absent from a document, the connector injects a stable value derived from the stream, topic, partition, offset, and message ID, avoiding Meilisearch primary-key inference failures. If your payloads carry their own primary key, its values must be unique. Otherwise add-or-replace semantics collapse distinct messages into one document.
With `include_metadata = true`, reserved `iggy_*` fields (stream, topic, partition, offset, checksum, timestamps) are written after payload parsing and overwrite same-named payload fields.
## Delivery Semantics
-The runtime commits consumer offsets when messages are polled and does not gate commits on `consume()`'s result, so the effective guarantee is **at-most-once** on sink errors; the retry settings provide best-effort retries within a single batch. Setting `wait_for_tasks = false` makes indexing fire-and-forget: submission succeeds before Meilisearch confirms indexing, and later task failures are neither observed nor retried.
+The runtime commits consumer offsets when messages are polled and doesn't gate commits on `consume()`'s result, so the effective guarantee is **at-most-once** on sink errors. The retry settings provide best-effort retries within a single batch. Setting `wait_for_tasks = false` makes indexing fire-and-forget: submission succeeds before Meilisearch confirms indexing, and later task failures are neither observed nor retried.
## Transforms
diff --git a/content/docs/connectors/sinks/s3.mdx b/content/docs/connectors/sinks/s3.mdx
index adcd73ea..1a77fdb0 100644
--- a/content/docs/connectors/sinks/s3.mdx
+++ b/content/docs/connectors/sinks/s3.mdx
@@ -53,7 +53,7 @@ output_format = "json_lines"
Further options cover credentials (`access_key_id` / `secret_access_key`), retries (`max_attempts` `3`, `retry_delay` `1s` with exponential backoff and jitter), and `path_style` addressing (auto-enabled when `endpoint` is set). See the upstream [s3_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/s3_sink) for the full reference.
-Path templates support `{stream}`, `{topic}`, `{partition}`, `{date}`, `{hour}`, and `{timestamp}`; the time-based variables derive from the first message timestamp in each buffer. File names embed the partition (zero-padded to 5 digits, to prevent cross-partition collisions) and the offset range (zero-padded to 20 digits), e.g. `application_logs/api_requests/2026-03-16/14/00000-00000000000000000000-00000000000000000999.jsonl`.
+Path templates support `{stream}`, `{topic}`, `{partition}`, `{date}`, `{hour}`, and `{timestamp}`. The time-based variables derive from the first message timestamp in each buffer. File names embed the partition (zero-padded to 5 digits, to prevent cross-partition collisions) and the offset range (zero-padded to 20 digits), e.g. `application_logs/api_requests/2026-03-16/14/00000-00000000000000000000-00000000000000000999.jsonl`.
## Credentials
diff --git a/content/docs/connectors/sinks/surrealdb.mdx b/content/docs/connectors/sinks/surrealdb.mdx
index 0aa63a12..f5f64cae 100644
--- a/content/docs/connectors/sinks/surrealdb.mdx
+++ b/content/docs/connectors/sinks/surrealdb.mdx
@@ -2,7 +2,7 @@
title: SurrealDB Sink
---
-The SurrealDB sink connector writes messages from Iggy streams into SurrealDB over the HTTP API. Each batch becomes one SurrealQL bulk `INSERT IGNORE`, and every record gets a deterministic record id derived from stream, topic, partition, offset, and message id, so replayed batches are idempotent and existing records are left untouched.
+The SurrealDB sink connector writes messages from Iggy streams into SurrealDB over the HTTP API. Each batch becomes one SurrealQL bulk `INSERT IGNORE`. Every record gets a deterministic record id derived from stream, topic, partition, offset, and message id, so replayed batches are idempotent and existing records are left untouched.
This page is a curated subset of the documentation. The canonical reference is the upstream [surrealdb_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/surrealdb_sink) in the `apache/iggy` repository.
@@ -61,7 +61,7 @@ With metadata enabled, each record contains the deterministic `id`, the `iggy_me
## Delivery Semantics
-The runtime commits consumer offsets before `consume()` runs, so persistent write failures are **at-most-once**: they are logged but not redelivered. Transient failures are retried inside the batch with capped exponential backoff. Because records use deterministic ids and `INSERT IGNORE`, a replayed batch (for example after a connector restart re-reads uncommitted messages) does not create duplicates.
+The runtime commits consumer offsets before `consume()` runs, so persistent write failures are **at-most-once**: they are logged but not redelivered. Transient failures are retried inside the batch with capped exponential backoff. Because records use deterministic ids and `INSERT IGNORE`, a replayed batch (for example after a connector restart re-reads uncommitted messages) doesn't create duplicates.
## Transforms
diff --git a/content/docs/connectors/sources/elasticsearch.mdx b/content/docs/connectors/sources/elasticsearch.mdx
index 1335328d..c7f21278 100644
--- a/content/docs/connectors/sources/elasticsearch.mdx
+++ b/content/docs/connectors/sources/elasticsearch.mdx
@@ -41,7 +41,7 @@ timestamp_field = "@timestamp"
| `batch_size` | integer | `100` | Maximum documents fetched per poll |
| `timestamp_field` | string | none | Document field used for incremental polling |
-A custom `query` must be a structured object. In TOML that means nested tables (an inline JSON string will not work):
+A custom `query` must be a structured object. In TOML that means nested tables (an inline JSON string won't work):
```toml
[plugin_config.query.term.level]
@@ -54,7 +54,7 @@ Alternatively, set `plugin_config_format = "json"` and provide the plugin config
Each cycle the connector waits `polling_interval`, then runs a search against `index` with the configured `query`, `size = batch_size`, sorted ascending by `timestamp_field` (`@timestamp` when unset). Every document's `_source` becomes one JSON message.
-When `timestamp_field` is set, the query is additionally filtered to documents whose timestamp is greater than the newest one seen so far, so already-processed documents are not fetched again. Without a `timestamp_field` there is no incremental cursor and each poll returns the first `batch_size` matching documents.
+When `timestamp_field` is set, the query is also filtered to documents whose timestamp is greater than the newest one seen so far, so already-processed documents are not fetched again. Without a `timestamp_field` there is no incremental cursor and each poll returns the first `batch_size` matching documents.
## State
diff --git a/content/docs/connectors/sources/influxdb.mdx b/content/docs/connectors/sources/influxdb.mdx
index f014efe7..b2623d61 100644
--- a/content/docs/connectors/sources/influxdb.mdx
+++ b/content/docs/connectors/sources/influxdb.mdx
@@ -2,7 +2,7 @@
title: InfluxDB Source
---
-The InfluxDB source connector polls InfluxDB on an interval and produces the resulting rows as messages into an Iggy stream. It supports both InfluxDB V2 (OSS 2.x / Cloud, Flux queries) and InfluxDB V3 (Core / Enterprise, SQL queries), selected with the `version` option, and tracks its position with a timestamp cursor stored in persistent connector state, so restarts resume where they left off.
+The InfluxDB source connector polls InfluxDB on an interval and produces the resulting rows as messages into an Iggy stream. It supports both InfluxDB V2 (OSS 2.x / Cloud, Flux queries) and InfluxDB V3 (Core / Enterprise, SQL queries), selected with the `version` option. A timestamp cursor stored in persistent connector state tracks the position, so restarts resume where they left off.
This page is a curated subset of the documentation. The canonical reference, including the full cursor semantics and stuck-timestamp handling, is the upstream [influxdb_source README](https://github.com/apache/iggy/tree/master/core/connectors/sources/influxdb_source) in the `apache/iggy` repository.
@@ -39,7 +39,7 @@ poll_interval = "5s"
batch_size = 500
```
-InfluxDB V2 takes an `org` and a Flux query; V3 takes a `db` and a SQL query:
+InfluxDB V2 takes an `org` and a Flux query. V3 takes a `db` and a SQL query:
```toml
# V3 variant
@@ -56,7 +56,7 @@ LIMIT $limit OFFSET $offset
'''
```
-Omitting `version` defaults to `"v2"` for backward compatibility. The query template must contain the `$cursor` placeholder (enforced at startup); `$limit` is strongly recommended so `batch_size` bounds each query. V3 queries additionally need `OFFSET $offset` unless stuck-batch detection is disabled.
+Omitting `version` defaults to `"v2"` for backward compatibility. The query template must contain the `$cursor` placeholder (enforced at startup). `$limit` is strongly recommended so `batch_size` bounds each query. V3 queries also need `OFFSET $offset` unless stuck-batch detection is disabled.
### Common Options
@@ -76,17 +76,17 @@ Omitting `version` defaults to `"v2"` for backward compatibility. The query temp
| `include_metadata` | bool | `true` | Include all row columns in the payload |
| `stuck_batch_cap_factor` | u32 | `10` | V3 only: cap on batch inflation for tied timestamps (max `100`, `0` disables) |
-Further options cover `payload_column` (extract a single column as the payload; required for `text` and `raw` formats) and resilience: `timeout` (`10s`), `max_retries` (`3`), `retry_delay` / `retry_max_delay`, startup health-check retries, and a circuit breaker (`circuit_breaker_threshold` `5`, `circuit_breaker_cool_down` `30s`). See the upstream [influxdb_source README](https://github.com/apache/iggy/tree/master/core/connectors/sources/influxdb_source) for the full list.
+Further options cover `payload_column` (extract a single column as the payload, required for `text` and `raw` formats) and resilience: `timeout` (`10s`), `max_retries` (`3`), `retry_delay` / `retry_max_delay`, startup health-check retries, and a circuit breaker (`circuit_breaker_threshold` `5`, `circuit_breaker_cool_down` `30s`). See the upstream [influxdb_source README](https://github.com/apache/iggy/tree/master/core/connectors/sources/influxdb_source) for the full list.
## Cursor-Based Polling
-On each poll the cursor (an RFC 3339 timestamp) is substituted into the query template; after a successful batch it advances to the highest timestamp seen. The two versions differ deliberately:
+On each poll the cursor (an RFC 3339 timestamp) is substituted into the query template. After a successful batch it advances to the highest timestamp seen. The two versions differ deliberately:
-- **V2** uses inclusive `>=` semantics, so the query must sort by the cursor field (`|> sort(columns: ["_time"])`); startup fails for `>=`-style queries lacking a sort. `range()`-style queries pass that check without a sort, but should keep it anyway, since the skip count below depends on ordered rows. Rows at the cursor timestamp are re-fetched after an advance, and the connector skips exactly the already-delivered ones using a persisted row count, preventing duplicates.
+- **V2** uses inclusive `>=` semantics, so the query must sort by the cursor field (`|> sort(columns: ["_time"])`). Startup fails for `>=`-style queries lacking a sort. `range()`-style queries pass that check without a sort, but should keep it anyway, since the skip count below depends on ordered rows. Rows at the cursor timestamp are re-fetched after an advance, and the connector skips exactly the already-delivered ones using a persisted row count, preventing duplicates.
- **V3** uses strict `WHERE time > '$cursor'`, so no rows are re-delivered across batches. Because InfluxDB 3's query engine has no stable order for rows sharing one timestamp, the connector pages through tied rows with `OFFSET $offset`, inflating the batch size up to `stuck_batch_cap_factor` times `batch_size` before tripping the circuit breaker.
## Payload Formats
-- **`json`** (default): each row becomes a JSON object. V3 emits flat rows with native JSON types; V2 wraps each row in an envelope (`measurement`, `field`, `timestamp`, `value`, `row`) with all values as strings coerced to bool/int/float where possible, since annotated CSV carries no types. A string field containing `"42"` arrives as the number `42` under V2 (coerced) but stays the string `"42"` under V3; values that cannot be coerced stay strings under V2. Migrating consumers must upda [...]
+- **`json`** (default): each row becomes a JSON object. V3 emits flat rows with native JSON types. V2 wraps each row in an envelope (`measurement`, `field`, `timestamp`, `value`, `row`) with all values as strings coerced to bool/int/float where possible, since annotated CSV carries no types. A string field containing `"42"` arrives as the number `42` under V2 (coerced) but stays the string `"42"` under V3. Values that can't be coerced stay strings under V2. Migrating consumers must updat [...]
- **`text`**: the value of `payload_column` is produced as a UTF-8 string.
- **`raw`**: the value of `payload_column` is base64-decoded and produced as raw bytes.
diff --git a/content/docs/connectors/sources/postgres.mdx b/content/docs/connectors/sources/postgres.mdx
index 8d32cc59..e41676be 100644
--- a/content/docs/connectors/sources/postgres.mdx
+++ b/content/docs/connectors/sources/postgres.mdx
@@ -88,7 +88,7 @@ By default (no `payload_column`), each row is wrapped in a JSON envelope:
}
```
-The stream config should use `schema = "json"`. Flat-schema sinks such as Iceberg need the `unwrap_envelope` transform to extract the `data` field; see the [transforms documentation](/docs/connectors/transforms).
+The stream config should use `schema = "json"`. Flat-schema sinks such as Iceberg need the `unwrap_envelope` transform to extract the `data` field. See the [transforms documentation](/docs/connectors/transforms).
### Payload Column Extraction
@@ -101,6 +101,6 @@ With `mode = "cdc"` the connector reads changes from a logical replication slot
- The builtin backend creates (or reuses) a logical replication slot using the `test_decoding` output plugin. The default slot name is `iggy_slot`. No publication is created or required, since `test_decoding` ignores publications entirely.
- PostgreSQL must run with `wal_level = logical`, and the connection must be direct (no pooler) where replication requires it.
- `capture_operations` filters which operations (`INSERT`, `UPDATE`, `DELETE`) are emitted.
-- The `pg_replicate` backend (based on Supabase's ETL framework) is a placeholder and is **not implemented**: selecting `cdc_backend = "pg_replicate"` fails at startup unless the `cdc_pg_replicate` build feature is enabled, and the backend returns an error even when it is.
+- The `pg_replicate` backend (based on Supabase's ETL framework) is **not implemented**: selecting `cdc_backend = "pg_replicate"` fails at startup unless the `cdc_pg_replicate` build feature is enabled, and the backend returns an error even when it is.
-When decommissioning a CDC connector, drop the replication slot to stop WAL retention; see the upstream README for details.
+When decommissioning a CDC connector, drop the replication slot to stop WAL retention. See the upstream README for details.
diff --git a/content/docs/connectors/sources/source.mdx b/content/docs/connectors/sources/source.mdx
index ba847ed8..0f8ec778 100644
--- a/content/docs/connectors/sources/source.mdx
+++ b/content/docs/connectors/sources/source.mdx
@@ -42,7 +42,7 @@ pub struct SourceConfig {
}
```
-The two flags at the end are optional and default to `false`: `verbose` switches the connector's per-batch logging to info level, and `benchmark` emits per-batch timing events for performance measurement.
+`verbose` and `benchmark` are optional and default to `false`. `verbose` switches the connector's per-batch logging to info level, and `benchmark` emits per-batch timing events for performance measurement.
**Main runtime config (config.toml):**
@@ -109,7 +109,7 @@ IGGY_CONNECTORS_SOURCE_RANDOM_PLUGIN_CONFIG_PAYLOAD_SIZE=200
## Sample implementation
-Let's implement the example source connector, which will simply generate a random batch of messages at the configured interval, optionally stopping after a total message limit.
+Let's implement the example source connector, which generates a random batch of messages at the configured interval and optionally stops after a total message limit.
Additionally, our source connector will have its own state, which can be used e.g. to track the overall progress or store some relevant information when producing the data from the actual external sources or tooling.
@@ -119,7 +119,7 @@ Also, when implementing the source connector, make sure to use the `source_conne
And finally, each source should have its own, custom configuration, which is passed along with the unique plugin ID and optional state via expected `new()` method.
-The full, compiling implementation lives in the repository at [core/connectors/sources/random_source](https://github.com/apache/iggy/tree/master/core/connectors/sources/random_source); the snippets below mirror it.
+The full, compiling implementation lives in the repository at [core/connectors/sources/random_source](https://github.com/apache/iggy/tree/master/core/connectors/sources/random_source). The snippets below mirror it.
Let's start by defining the internal state and the public source connector along with its own configuration.
@@ -202,7 +202,7 @@ At a bare minimum, we need to add the following dependencies to the `Cargo.toml`
- tokio
- tracing
-This particular example additionally uses `humantime`, `rand`, `rmp-serde`, `simd-json`, and `uuid`.
+This example also uses `humantime`, `rand`, `rmp-serde`, `simd-json`, and `uuid`.
Before we make use of the `Source` trait, let's define the internal payload of the message that will be produced (e.g. as if it was pulled from some external database or so).
@@ -216,7 +216,7 @@ struct Record {
}
```
-Next, a few helpers: `serialize_state()` encodes the state to MessagePack via the `ConnectorState::serialize()` helper, and `generate_messages()` builds a random number of messages (within `messages_range`), each carrying a JSON-serialized `Record` with `payload_size` random characters of text.
+`serialize_state()` encodes the state to MessagePack via `ConnectorState::serialize()`. `generate_messages()` builds a random number of messages (within `messages_range`), each carrying a JSON-serialized `Record` with `payload_size` random characters of text.
```rust
impl RandomSource {
@@ -267,7 +267,7 @@ impl RandomSource {
}
```
-Now, let's implement the `Source` trait for our `RandomSource` struct. Each `poll()` waits for the configured `interval` to mimic the behavior of a real-world external source, stops producing once `max_count` messages were generated (when it is set), and returns the updated state along with the `ProducedMessages` - the runtime persists that state and hands it back to `new()` after a restart.
+Now, let's implement the `Source` trait for our `RandomSource` struct. Each `poll()` waits for the configured `interval` to mimic a real-world external source, stops producing once `max_count` messages have been generated (if it's set), and returns the updated state along with the `ProducedMessages`. The runtime persists that state and hands it back to `new()` after a restart.
```rust
#[async_trait]
diff --git a/content/docs/connectors/transforms.mdx b/content/docs/connectors/transforms.mdx
index 4f480c79..a33548c2 100644
--- a/content/docs/connectors/transforms.mdx
+++ b/content/docs/connectors/transforms.mdx
@@ -2,7 +2,7 @@
title: Transforms
---
-Transforms mutate, filter, or convert messages as they flow through the connector runtime. They run inside the runtime process, not inside the plugins: for source connectors they are applied after the plugin's messages are decoded and before producing to Iggy, and for sink connectors after consuming from Iggy and before handing the messages to the plugin.
+Transforms mutate, filter, or convert messages as they flow through the connector runtime. They run inside the runtime process, not inside the plugins. For source connectors they're applied after the plugin's messages are decoded and before producing to Iggy. For sink connectors, after consuming from Iggy and before handing the messages to the plugin.
A transform receives each decoded message and returns either the (possibly modified) message, or nothing - in which case the message is dropped from the batch.
@@ -26,9 +26,9 @@ fields = ["internal_id"]
Two rules apply to every transform:
- The `enabled` key is required. A transform section without it fails to load, and a transform with `enabled = false` is skipped.
-- The order of the sections in the file does not determine the execution order - do not rely on one transform seeing the output of another.
+- The order of the sections in the file doesn't determine the execution order, so don't rely on one transform seeing the output of another.
-The field-level transforms (`add_fields`, `delete_fields`, `filter_fields`, `update_fields`, `unwrap_envelope`) operate on JSON payloads only; messages with any other payload format pass through them unchanged.
+The field-level transforms (`add_fields`, `delete_fields`, `filter_fields`, `update_fields`, `unwrap_envelope`) operate on JSON payloads only. Messages with any other payload format pass through them unchanged.
## add_fields
@@ -108,8 +108,8 @@ value_pattern = "is_not_null"
```
- **`keep_fields`**: fields that are always kept, regardless of the patterns.
-- **`patterns`**: a list of patterns; a field matches when any pattern matches. Within one pattern, `key_pattern` and `value_pattern` must both match when both are set (an omitted one matches everything).
-- **`include_matching`**: `true` (default) keeps the matching fields and drops the rest; `false` drops the matching fields and keeps the rest.
+- **`patterns`**: a list of patterns. A field matches when any pattern matches. Within one pattern, `key_pattern` and `value_pattern` must both match when both are set (an omitted one matches everything).
+- **`include_matching`**: `true` (default) keeps the matching fields and drops the rest. `false` drops the matching fields and keeps the rest.
`key_pattern` variants: `exact`, `starts_with`, `ends_with`, `contains`, `regex` - each takes a string, e.g. `key_pattern = { contains = "name" }`.
@@ -125,7 +125,7 @@ enabled = true
field = "data"
```
-`field` must not be empty. Messages that are not JSON objects, or that do not contain the field, pass through unchanged.
+`field` must not be empty. Messages that aren't JSON objects, or that don't contain the field, pass through unchanged.
## proto_convert
@@ -194,7 +194,7 @@ strict_mode = false
Optional keys: `schema_path`, `schema_json`, `field_mappings`.
-Note that unlike the field-level transforms, the three format-conversion transforms define no per-key defaults: every key shown in their examples (except the ones listed as optional) must be present, or the connector configuration fails to load. The `source_format` and `target_format` keys accept any schema value: `json`, `raw`, `text`, `proto`, `flat_buffer`, `avro`. The FlatBuffers transform additionally restricts the pair: only JSON to/from FlatBuffers, FlatBuffers to Text or Raw, and [...]
+Unlike the field-level transforms, the three format-conversion transforms define no per-key defaults: every key shown in their examples (except the ones listed as optional) must be present, or the connector configuration fails to load. The `source_format` and `target_format` keys accept any schema value: `json`, `raw`, `text`, `proto`, `flat_buffer`, `avro`. The FlatBuffers transform additionally restricts the pair: only JSON to/from FlatBuffers, FlatBuffers to Text or Raw, and identity [...]
## Adding a new transform
diff --git a/content/docs/faq/faq.mdx b/content/docs/faq/faq.mdx
index 9779f1f8..26764ac2 100644
--- a/content/docs/faq/faq.mdx
+++ b/content/docs/faq/faq.mdx
@@ -24,7 +24,7 @@ When consumers join or leave a consumer group, the server triggers cooperative p
## Q: Does Iggy support exactly-once delivery?
-Iggy supports **at-most-once** (with auto-commit) and **at-least-once** (without auto-commit, manual offset management). **Exactly-once** semantics can be achieved at the application level, e.g. by attaching unique message IDs to messages and deduplicating on the consumer side.
+Iggy supports **at-most-once** (with auto-commit) and **at-least-once** (without auto-commit, manual offset management). **Exactly-once** semantics can be achieved at the application level, e.g. by attaching unique IDs to messages and deduplicating on the consumer side.
## Q: How do I secure my Iggy deployment?
@@ -40,7 +40,7 @@ Iggy migrated from Tokio to a thread-per-core architecture with `compio` (which
## Q: Is clustering/replication available?
-Yes. Replication based on Viewstamped Replication (VSR) is built into the server. Multi-node clustering is configured via the `[cluster]` section (single-node mode is the default, `enabled = false`), covers consensus, view changes, node authentication, and TLS between nodes, and is maturing toward production readiness. See the [Clustering](/docs/clustering/vsr) documentation for the protocol and [Deploy](/docs/clustering/deploy) for setting up a cluster.
+Yes. Replication based on Viewstamped Replication (VSR) is built into the server. Multi-node clustering is configured via the `[cluster]` section (single-node mode is the default, `enabled = false`). It covers consensus, view changes, node authentication, and TLS between nodes, and is maturing toward production readiness. See the [Clustering](/docs/clustering/vsr) documentation for the protocol and [Deploy](/docs/clustering/deploy) for setting up a cluster.
## Q: What examples are available?
diff --git a/content/docs/index.mdx b/content/docs/index.mdx
index a43f2f56..3ad40fcc 100644
--- a/content/docs/index.mdx
+++ b/content/docs/index.mdx
@@ -22,7 +22,7 @@ Every partition is an append-only log. Messages are **immutable** once written,
| Capability | Details |
|-----------|---------|
-| **Transport** | TCP, QUIC, WebSocket, HTTP with custom binary protocol |
+| **Transport** | TCP, QUIC, WebSocket (custom binary protocol), HTTP (REST) |
| **Performance** | Thread-per-core + io_uring, zero-copy deserialization, custom 4 GiB memory pool |
| **Security** | TLS on all transports, Argon2id hashing, AES-256-GCM encryption, [granular RBAC](/docs/server/security) with per-stream/per-topic permissions |
| **Connectors** | [14 sink and 4 source plugins](/docs/connectors/introduction) including PostgreSQL, MongoDB, Elasticsearch, ClickHouse, Apache Iceberg, S3, with data transforms |
diff --git a/content/docs/introduction/about.mdx b/content/docs/introduction/about.mdx
index ef8f9f47..46a109df 100644
--- a/content/docs/introduction/about.mdx
+++ b/content/docs/introduction/about.mdx
@@ -87,7 +87,7 @@ Features include stream/topic/partition management, message browser with JSON/st
Iggy provides a highly performant and modular **[runtime](/docs/connectors/runtime)** for statically typed, yet dynamically loaded connectors. You can ingest data from external sources and push the data to Iggy streams, or fetch data from Iggy streams and forward it to external systems. **Create your own Rust plugins** by simply implementing either the `Source` or `Sink` trait and **build custom pipelines for the data processing**.
-There are currently 14 sinks and 4 sources available. Sinks include PostgreSQL, MongoDB, Elasticsearch, ClickHouse, InfluxDB, Apache Iceberg, Delta Lake, Quickwit and S3; sources include PostgreSQL, Elasticsearch, InfluxDB and random data generation. See the [connectors documentation](/docs/connectors/introduction) for the full catalog.
+There are currently 14 sinks and 4 sources available. Sinks include PostgreSQL, MongoDB, Elasticsearch, ClickHouse, InfluxDB, Apache Iceberg, Delta Lake, Quickwit and S3. Sources include PostgreSQL, Elasticsearch, InfluxDB and random data generation. See the [connectors documentation](/docs/connectors/introduction) for the full catalog.
The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect`.
diff --git a/content/docs/introduction/architecture.mdx b/content/docs/introduction/architecture.mdx
index 37c456d9..e453ca53 100644
--- a/content/docs/introduction/architecture.mdx
+++ b/content/docs/introduction/architecture.mdx
@@ -34,7 +34,7 @@ All shards, **including shard 0**, own partitions and serve partition requests.
Requests are routed between shards using **message passing** (via `crossfire` bounded mpsc channels), which avoids locking entirely. The routing logic splits operations into two planes:
- **Metadata operations** (create/delete stream/topic/user etc.) always execute on **shard 0** - it is the only shard that commits metadata
-- **Partition operations** (send_messages, poll_messages, store_consumer_offset) are routed to the shard owning that partition via a lock-free concurrent map lookup; a request that lands on a non-owning shard rides the inter-shard message bus to the owner
+- **Partition operations** (send_messages, poll_messages, store_consumer_offset) are routed to the shard owning that partition via a lock-free concurrent map lookup. A request that lands on a non-owning shard rides the inter-shard message bus to the owner
Partition-to-shard assignment is **deterministic**: the packed `IggyNamespace` is hashed with `Murmur3`, and the upper 16 bits of the hash are taken modulo the shard count (the upper bits are used because Murmur3 has weak lower bits for small integer inputs).
@@ -107,7 +107,7 @@ local_data/
└── offsets/
```
-The stream, topic and partition directories are named after their numeric IDs, **assigned from 0**. The `metadata/journal.wal` file is the VSR write-ahead log that persists all metadata operations (stream/topic/user creation, etc.). The `runtime/current_config.toml` file captures the configuration the server actually booted with. Segment files are named by the 20-digit start offset of their first record; the `.index` file is created automatically and is used to speed up the search operat [...]
+The stream, topic and partition directories are named after their numeric IDs, **assigned from 0**. The `metadata/journal.wal` file is the VSR write-ahead log that persists all metadata operations (stream/topic/user creation, etc.). The `runtime/current_config.toml` file captures the configuration the server actually booted with. Segment files are named by the 20-digit start offset of their first record. The `.index` file is created automatically and speeds up searches by keeping track o [...]
## Memory pool
diff --git a/content/docs/introduction/concepts.mdx b/content/docs/introduction/concepts.mdx
index 07862ad4..c0d74223 100644
--- a/content/docs/introduction/concepts.mdx
+++ b/content/docs/introduction/concepts.mdx
@@ -2,27 +2,25 @@
title: Concepts
---
-Iggy is the persistent message streaming, which means, that the messages are being stored in a form of an **append-only log**. You can create multiple streams, consisting of topics, which might have one or more partitions assigned, e.g. to achieve the horizontal scalability between many independent consumers or higher system resiliency. You can think of Iggy as an alternative to Kafka or RabbitMQ streams.
-
-Let's discuss in-depth what these concepts are all about and how they relate to each other.
+Iggy is a persistent message streaming platform: messages are stored in a form of an **append-only log**. You can create multiple streams, consisting of topics, which might have one or more partitions assigned, e.g. to achieve the horizontal scalability between many independent consumers or higher system resiliency. You can think of Iggy as an alternative to Kafka or RabbitMQ streams.
## Message streaming
-There's a high chance that you've already used messaging tools such as RabbitMQ or Kafka, just to name a few. While they might look similar at the first glance, and actually, you can achieve the similar results with all of them (e.g. publishing and consuming the events by the different applications built on top of microservices architecture), they are quite different in their core.
+You've probably used RabbitMQ or Kafka already. They look similar at the first glance, and you can achieve the similar results with both (e.g. publishing and consuming the events by the different applications built on top of microservices architecture), but they work differently underneath.
The main difference is that RabbitMQ (except the recently released Streams plugin) is the **message broker**, which means that it's responsible for delivering the messages to the consumers. It works in the FIFO (First In, First Out) manner and the messages are being kept in the queues. For example, if you have multiple, distinct consumers, then each one would create its own queue, the message would be replicated between each queue and each consumer would be responsible for reading the me [...]
On the other hand, Kafka is a **message streaming platform**, meaning that it's not responsible for delivering the messages to the consumers, but rather it's storing them in a form of an append-only log. The consumers are responsible for reading the messages from the log and processing them. You might have multiple distinct consumers, and it doesn't affect the resource usage as there's only one log. The consumers can read the messages from the beginning, or from the specific offset, thus [...]
-There are advantages and disadvantages of both approaches, but the main difference is that the message broker is responsible for delivering the messages to the consumers, while the message streaming platform is not. The message broker is a more mature concept, but the message streaming platform is gaining more and more popularity, especially in the cloud-native world. And you can achieve much higher performance and throughput with the message streaming platform, since it acts as a simple [...]
+Both approaches have advantages and disadvantages. The message broker is a more mature concept, but the message streaming platform is gaining more and more popularity, especially in the cloud-native world. And you can achieve much higher performance and throughput with the message streaming platform, since it acts as a simple database, being optimized for the append-only operations and can be queried in a very efficient way.
-As you might've guessed by now, Iggy is the latter - the message streaming platform.
+Iggy is the latter, a message streaming platform.
## Append-only log
The append-only log is the core concept of Iggy. It's a simple data structure, which is optimized for the append-only operations. It's a sequence of records, that are being appended to the end of the log. The records are **immutable**, so that they can't be changed once they are written to the log. The records are being written in the order they are received, which results in the log being ordered.
-To navigate the log, you can use the **offset**, which is the position of the record in the log. The offset is a simple integer, that starts from 0 and is incremented by 1 for each record. When the client is reading the records from the log, it can specify the offset from which it wants to start reading the records. The client can also specify the maximum number of records it wants to read. The client can read the records from the beginning, or from the specific offset, which means that [...]
+You address the log by **offset**, the position of the record in the log. The offset is a simple integer that starts from 0 and is incremented by 1 for each record. When the client reads the records, it specifies the offset to start from and the maximum number of records it wants. Starting from the beginning, or from any earlier offset, is how you replay the messages.
<AppendOnlyLogViz />
@@ -37,7 +35,7 @@ The stream is a logical concept, and you might think of it as a **namespace**. F
The topic is also the logical concept, which is a part of the stream. The topic is identified by its unique ID. You could think of topic as an entity being responsible for storing the specific type of the records. For example, you could have a topic for the user events, and another topic for the order events, etc.
-The important thing to note is that, the messages are not being stored in the topic directly, but rather in the **partitions**, which are assigned to the topic. The topic can have one or more partitions assigned, that could help achieve higher parallelism and throughput. The topic can also have the **retention policy** assigned, which means that the records are being deleted automatically once they are older than the specified retention period. Topics also support configurable compressio [...]
+The messages are not being stored in the topic directly, but rather in the **partitions**, which are assigned to the topic. The topic can have one or more partitions assigned, that could help achieve higher parallelism and throughput. The topic can also have the **retention policy** assigned, which means that the records are being deleted automatically once they are older than the specified retention period. Topics also support configurable compression (`none`, `gzip`), maximum size limi [...]
## Partition
@@ -61,10 +59,10 @@ Each segment consists of:
Consumers can poll the messages in multiple ways:
-- **By offset** - start reading from the specified offset; the client tracks its own position.
+- **By offset** - start reading from the specified offset. The client tracks its own position.
- **By timestamp** - start reading from the first message at or after the given timestamp.
- **First / Last** - start from the beginning or the end of the partition.
-- **Next** - continue from the consumer offset stored on the server side; the client no longer needs to track the offset itself. Combine it with `auto_commit: true` to commit the offset automatically once the messages are fetched (*at-most-once* delivery), or call `store_offset()` explicitly after processing.
+- **Next** - continue from the consumer offset stored on the server side. The client no longer needs to track the offset itself. Combine it with `auto_commit: true` to commit the offset automatically once the messages are fetched (*at-most-once* delivery), or call `store_offset()` explicitly after processing.
## Consumer groups
@@ -76,7 +74,7 @@ Consumer groups provide horizontal scaling for message consumption. When multipl
<MessageHeaderDiagram />
-In the SDKs, each message carries a 64-byte in-memory header (little-endian fields). On the wire and on disk, messages travel inside batch records with a compact 48-byte per-message frame; see the [binary schema](/docs/server/schema) for the exact encodings.
+In the SDKs, each message carries a 64-byte in-memory header (little-endian fields). On the wire and on disk, messages travel inside batch records with a compact 48-byte per-message frame. See the [binary schema](/docs/server/schema) for the exact encodings.
| Field | Bytes | Type | Description |
|-------|-------|------|-------------|
diff --git a/content/docs/introduction/getting-started.mdx b/content/docs/introduction/getting-started.mdx
index ad14cf4d..7d012e12 100644
--- a/content/docs/introduction/getting-started.mdx
+++ b/content/docs/introduction/getting-started.mdx
@@ -39,7 +39,7 @@ Alternatively, build from source by cloning the [repository](https://github.com/
cargo run --bin iggy-server -- --fresh --with-default-root-credentials
```
-A word about credentials: a bare `iggy-server` boot does **not** create the root user with the well-known `iggy`/`iggy` pair. It generates a random password and prints it **exactly once** to the logs. The `--with-default-root-credentials` flag switches to the development credentials (username: `iggy`, password: `iggy`), and `--fresh` wipes the data directory first, so the root user is created anew with them. You can also set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` yourself; the env [...]
+A bare `iggy-server` boot does **not** create the root user with the well-known `iggy`/`iggy` pair. It generates a random password and prints it **exactly once** to the logs. The `--with-default-root-credentials` flag switches to the development credentials (username: `iggy`, password: `iggy`), and `--fresh` wipes the data directory first, so the root user is created anew with them. You can also set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` yourself. The environment **takes precedenc [...]
All the data used by the server will be persisted under the `local_data` directory, unless specified differently in the configuration.
@@ -117,7 +117,7 @@ We could make use of more advanced components such as [`ClientProvider`](https:/
If you're eager to find out how to build more advanced (and configurable) applications, check the Rust [examples](/docs/sdk/rust/examples). Nevertheless, let's focus on implementing our producer side :)
-Next, we need to authenticate the user, as all the available actions (except the `ping` and `login`) require the user to be authenticated and have the appropriate permissions e.g. you might be able to send or poll the messages, but you won't be able to create the stream or topic. For the sake of simplicity, we will use the development credentials for the root user (username: `iggy`, password: `iggy`), that can do anything and cannot be deleted. These credentials exist only because we sta [...]
+Next, we need to authenticate the user, as all the available actions (except the `ping` and `login`) require the user to be authenticated and have the appropriate permissions e.g. you might be able to send or poll the messages, but you won't be able to create the stream or topic. For the sake of simplicity, we will use the development credentials for the root user (username: `iggy`, password: `iggy`), that can do anything and cannot be deleted. These credentials exist only because we sta [...]
```rust
client
@@ -153,7 +153,7 @@ So far, so good, however, before we will be able to publish any messages to our
Since our `IggyClient` implements the common [Client](https://github.com/apache/iggy/blob/master/core/sdk/src/clients/client.rs) trait, you can find lots of the different methods to interact with the server, also from the administrative point of view, e.g. creating the streams, topics etc.
-One thing worth mentioning is that, these methods are **not idempotent** - for example, if you were to try creating the stream with the same name which already exists on the server, you would receive the specific error. In such a case, you can simply check for an error and move on. When creating a stream, we only provide its name - the server assigns the numeric ID automatically. Stream, topic and partition IDs are all assigned **starting from 0**. Let's do this then :)
+These methods are **not idempotent** - for example, if you were to try creating the stream with the same name which already exists on the server, you would receive the specific error. In such a case, you can simply check for an error and move on. When creating a stream, we only provide its name - the server assigns the numeric ID automatically. Stream, topic and partition IDs are all assigned **starting from 0**. Let's do this then :)
```rust
use iggy::prelude::*;
@@ -199,7 +199,7 @@ async fn init_system(client: &IggyClient) {
}
```
-Every field of `TopicCreateOptions` left as `None` resolves to the **server defaults**. It is also where the per-topic durability knobs live: `segment_size`, `enforce_fsync` and the flush thresholds (`messages_required_to_save`, `size_of_messages_required_to_save`) can all be set at topic creation.
+Every field of `TopicCreateOptions` left as `None` resolves to the **server defaults**. It's also where the per-topic durability knobs live: `segment_size`, `enforce_fsync` and the flush thresholds (`messages_required_to_save`, `size_of_messages_required_to_save`) can all be set at topic creation.
Finally, let's send some messages into our stream. We will implement the basic loop with an interval between each iteration to simulate publishing the batch of messages. Since the streaming server works directly with the binary data, and couldn't care less about the (de)serialization format for the message payload, it's really up to you, how to efficiently stream the messages for your use case.
@@ -253,7 +253,7 @@ pub struct Partitioning {
In our scenario, we simply make use of `PartitioningKind::PartitionId` (by invoking the helper method `partition_id()`) meaning that we want to send the messages to the specified partition ID - in that case, the length will be 4 bytes (as the partition ID is of type `u32`), and the value will be the actual partition ID encoded as the bytes array.
-However, once your system grows, you might want to parallelize the messages across the independent consumers, in order to achieve the horizontal scaling, higher resiliency etc. In that case, you might consider using either `PartitioningKind::Balanced` (the server picks the next partition using the simple round-robin algorithm approach e.g. 0->1->2->0->1->2 etc.) or `PartitioningKind::MessagesKey` instead (e.g. by invoking one of the helper methods `messages_key()`), where the value would [...]
+However, once your system grows, you might want to parallelize the messages across the independent consumers, in order to achieve the horizontal scaling, higher resiliency etc. In that case, you might consider using either `PartitioningKind::Balanced` (the SDK picks the next partition using a client-local round-robin e.g. 0->1->2->0->1->2 etc.) or `PartitioningKind::MessagesKey` instead (e.g. by invoking one of the helper methods `messages_key()`), where the value wouldn't be a partition [...]
For example, given that you process the set of messages related to the specific order ID (e.g. created, confirmed, paid, delivered etc.), you could use that as a key value to ensure that all the messages which are part of the specific workflow, will always be put onto the same partition. The value of the key can be anything (e.g. string, number) with the **maximum length of 255 bytes**.
diff --git a/content/docs/sdk/connection-strings.mdx b/content/docs/sdk/connection-strings.mdx
index 223a7504..4f72e32a 100644
--- a/content/docs/sdk/connection-strings.mdx
+++ b/content/docs/sdk/connection-strings.mdx
@@ -44,7 +44,7 @@ iggy://username:password@localhost:8090
iggy://<personal-access-token>@localhost:8090
```
-**Do not prepend anything** to the token. Strings like `iggypat-...` appear only in Iggy's own test code; a real token with a prefix added will fail to log in.
+**Do not prepend anything** to the token. Strings like `iggypat-...` appear only in Iggy's own test code. A real token with a prefix added will fail to log in.
The default `iggy`/`iggy` root credentials **only exist** when the server was started with `--with-default-root-credentials` (or with `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` set) on its first boot. Otherwise the root user gets a generated password and samples using `iggy:iggy` fail with `InvalidCredentials`:
@@ -75,7 +75,7 @@ iggy://iggy:iggy@localhost:8090?tls=true&tls_domain=example.com&reconnection_ret
## QUIC options
-Note that the QUIC reconnection keys **differ from TCP**: `reconnection_max_retries` (not `reconnection_retries`) and `reconnection_reestablish_after` (not `reestablish_after`). Using the TCP names in a QUIC string is a hard error.
+The QUIC reconnection keys **differ from TCP**: `reconnection_max_retries` (not `reconnection_retries`) and `reconnection_reestablish_after` (not `reestablish_after`). Using the TCP names in a QUIC string is a hard error.
| Key | Description | Default |
|-----|-------------|---------|
diff --git a/content/docs/sdk/cpp/intro.mdx b/content/docs/sdk/cpp/intro.mdx
index 4f253902..541575e9 100644
--- a/content/docs/sdk/cpp/intro.mdx
+++ b/content/docs/sdk/cpp/intro.mdx
@@ -2,13 +2,13 @@
title: C++ SDK (WIP)
---
-The Iggy C++ SDK wraps the Rust SDK through a [CXX](https://cxx.rs) FFI bridge, which means it can access all transport protocols (TCP, QUIC, HTTP, WebSocket) through connection strings. The source code is available at [github.com/apache/iggy/tree/master/foreign/cpp](https://github.com/apache/iggy/tree/master/foreign/cpp).
+The Iggy C++ SDK wraps the Rust SDK through a [CXX](https://cxx.rs) FFI bridge, so it can use all transport protocols (TCP, QUIC, HTTP, WebSocket) via connection strings. The source code is available at [github.com/apache/iggy/tree/master/foreign/cpp](https://github.com/apache/iggy/tree/master/foreign/cpp).
-This SDK is not published to any package registry. It is consumed from the monorepo and built from source with Bazel.
+This SDK isn't published to any package registry. Consume it from the monorepo and build it from source with Bazel.
## Building from source
-The project builds with Bazel (`BUILD.bazel` / `MODULE.bazel`). The Bazel build drives the system-provided cargo toolchain to compile the Rust bridge, so concurrent runs can race; run builds serially.
+The project builds with Bazel (`BUILD.bazel` / `MODULE.bazel`). The build drives the system-provided cargo toolchain to compile the Rust bridge, so concurrent runs can race. Run builds serially.
```bash
cd foreign/cpp
diff --git a/content/docs/sdk/csharp/examples.mdx b/content/docs/sdk/csharp/examples.mdx
index d9518baf..427922e3 100644
--- a/content/docs/sdk/csharp/examples.mdx
+++ b/content/docs/sdk/csharp/examples.mdx
@@ -101,7 +101,7 @@ Run several instances of the consumer to see the group load-balance partitions b
## Typed messages
-A typed publisher/consumer pair that (de)serializes a record as JSON. Note that the typed builders are configured via statements rather than one fluent chain - the `With*` methods return the untyped base builder, so `Build()` **must be called on the typed builder variable** (see [Typed consumer](/docs/sdk/csharp/high-level-sdk#typed-consumer)):
+A typed publisher/consumer pair that (de)serializes a record as JSON. The typed builders are configured via statements rather than one fluent chain - the `With*` methods return the untyped base builder, so `Build()` **must be called on the typed builder variable** (see [Typed consumer](/docs/sdk/csharp/high-level-sdk#typed-consumer)):
```csharp
using System.Buffers;
diff --git a/content/docs/sdk/csharp/guide.mdx b/content/docs/sdk/csharp/guide.mdx
index 89135d3b..d6b5d900 100644
--- a/content/docs/sdk/csharp/guide.mdx
+++ b/content/docs/sdk/csharp/guide.mdx
@@ -99,7 +99,7 @@ The TCP client pings the server on its own every `HeartbeatInterval` while conne
| `UseExponentialBackoff` | `bool` | `true` | Use exponential backoff for delays |
| `BackoffMultiplier` | `double` | `2.0` | Multiplier for exponential backoff |
-> **Warning:** Reconnection is **on by default** with **unlimited retries**. Only a failed dial is retried; a rejected certificate, bad credentials, or a missing leader is thrown right away. With the default `MaxRetries = 0` an unreachable server is retried forever, so a request that passes no `CancellationToken` waits for as long as the server stays down. Set `MaxRetries` or pass a token to bound the wait, and set `Enabled = false` to opt out of reconnection entirely. Reconnection only [...]
+> **Warning:** Reconnection is **on by default** with **unlimited retries**. Only a failed dial is retried. A rejected certificate, bad credentials, or a missing leader is thrown right away. With the default `MaxRetries = 0` an unreachable server is retried forever, so a request that passes no `CancellationToken` waits for as long as the server stays down. Set `MaxRetries` or pass a token to bound the wait, and set `Enabled = false` to opt out of reconnection entirely. Reconnection only [...]
#### `AutoLoginSettings`
@@ -110,7 +110,7 @@ The TCP client pings the server on its own every `HeartbeatInterval` while conne
| `Password` | `string` | `""` | Password for auto-login |
| `PersonalAccessToken` | `string` | `""` | Personal access token to sign in with instead of a username and password; takes precedence over them when set |
-All properties are init-only: build the settings with an object initializer or the static factories `AutoLoginSettings.For(username, password)` and `AutoLoginSettings.ForPersonalAccessToken(token)`. Configuring auto-login is what enables transparent request replay after a reconnect; without stored credentials a reconnect **cannot restore the session**, and requests fail until the client logs in again.
+All properties are init-only: build the settings with an object initializer or the static factories `AutoLoginSettings.For(username, password)` and `AutoLoginSettings.ForPersonalAccessToken(token)`. Configuring auto-login is what enables transparent request replay after a reconnect. Without stored credentials a reconnect **cannot restore the session**, and requests fail until the client logs in again.
## Message encryption
@@ -344,7 +344,7 @@ await client.CreateTopicAsync(
A property left `null` emits no key and keeps the server default. These keys are **settable at creation only**: `UpdateTopicAsync` refuses them by name, the same way it refuses a key outside the server's catalog, so a mistyped key fails the call instead of being silently ignored. Use [`DescribeOptionsAsync`](#option-catalog) to enumerate the keys a server accepts.
-There is **no on-demand flush command**; durability is configured per topic through `EnforceFsync` and the flush thresholds above.
+There is **no on-demand flush command**. Durability is configured per topic through `EnforceFsync` and the flush thresholds above.
```csharp
var topicId = Identifier.String("my-topic");
@@ -424,7 +424,7 @@ Partitioning.EntityIdGuid(Guid.NewGuid())
Partitioning.EntityIdBytes(new byte[] { 1, 2, 3 })
```
-Partition selection is resolved **client-side** on both transports; the server never routes. Balanced partitioning is a client-side round-robin over the topic's partitions, and key-based routing hashes the key with xxHash32 modulo the partition count, consistent with the Rust SDK.
+Partition selection is resolved **client-side** on both transports. Balanced partitioning round-robins over the topic's partitions, and key-based routing hashes the key with xxHash32 modulo the partition count, consistent with the Rust SDK.
### User-defined headers
diff --git a/content/docs/sdk/csharp/high-level-sdk.mdx b/content/docs/sdk/csharp/high-level-sdk.mdx
index 6bf2ea34..f4297cc9 100644
--- a/content/docs/sdk/csharp/high-level-sdk.mdx
+++ b/content/docs/sdk/csharp/high-level-sdk.mdx
@@ -84,7 +84,7 @@ await publisher.DisposeAsync();
| `flushInterval` | `TimeSpan?` | `100ms` | Interval at which pending messages are flushed |
| `disposalTimeout` | `TimeSpan?` | `5s` | How long `DisposeAsync` waits for the background processor to drain |
-A batch also flushes once its accumulated payload reaches `IggyPublisherConfig.BackgroundMaxBatchBytes` (default 256 KB, `0` disables the byte gate) - whichever of `batchSize` or the byte limit is hit first. This knob has no builder method; set it on the config directly if needed.
+A batch also flushes once its accumulated payload reaches `IggyPublisherConfig.BackgroundMaxBatchBytes` (default 256 KB, `0` disables the byte gate) - whichever of `batchSize` or the byte limit is hit first. This knob has no builder method. Set it on the config directly if needed.
### Publishing with rented batches
diff --git a/content/docs/sdk/csharp/intro.mdx b/content/docs/sdk/csharp/intro.mdx
index 1b638db7..04b57df8 100644
--- a/content/docs/sdk/csharp/intro.mdx
+++ b/content/docs/sdk/csharp/intro.mdx
@@ -12,35 +12,35 @@ The SDK is built around the `IIggyClient` interface, which aggregates every feat
dotnet add package Apache.Iggy --prerelease
```
-The SDK targets .NET 8 and .NET 10 (`net8.0` and `net10.0`). Stable versions on NuGet stop at `0.8.0`, which predates the current wire protocol and cannot talk to a current server. The `0.9.0` line ships as **prerelease versions** (`0.9.0-edge.x` at the time of writing), so `--prerelease` is required, not optional.
+The SDK targets .NET 8 and .NET 10 (`net8.0` and `net10.0`). Stable versions on NuGet stop at `0.8.0`, which predates the current wire protocol and cannot talk to a current server. The `0.9.0` line ships as **prerelease versions** (`0.9.0-edge.x` at the time of writing), so `--prerelease` is required.
## Supported protocols
The SDK supports two transport protocols:
-- **TCP** - binary protocol for optimal performance and lower latency (recommended). Over TCP the SDK speaks the VSR consensus framing, the only wire protocol the server accepts; see [Connection semantics](#connection-semantics)
+- **TCP** - binary protocol for optimal performance and lower latency (recommended). Over TCP the SDK speaks the VSR consensus framing, the only wire protocol the server accepts. See [Connection semantics](#connection-semantics)
- **HTTP** - RESTful JSON API for stateless operations. `BaseAddress` must then be a full URI such as `http://127.0.0.1:3000` (the server's default HTTP port is 3000, TCP is 8090)
Some operations are **TCP-only** and throw `FeatureUnavailableException` on HTTP: joining/leaving a consumer group, `GetMeAsync`, `DeleteSegmentsAsync`, and `SendBinaryRequestAsync`.
## Connection semantics
-Over TCP every request is wrapped in a 256-byte consensus header (Viewstamped Replication), the client registers a consensus session at login, and writes are replicated before they are acknowledged. The `IIggyClient` surface is unchanged; what changes is the behavior around connections, groups, and failures:
+Over TCP every request is wrapped in a 256-byte consensus header (Viewstamped Replication), the client registers a consensus session at login, and writes are replicated before they are acknowledged. The `IIggyClient` surface is unchanged. What changes is the behavior around connections, groups, and failures:
- **Login binds a session.** `LoginUserAsync` / `LoginWithPersonalAccessTokenAsync` run the register handshake, and the session lives for as long as the connection. Logging out, being evicted, or losing the connection ends it, and the next login registers a fresh one.
- **Leader redirection is automatic.** The client reads the cluster roster, follows the current leader, and re-checks it when a request is refused because the node stopped being primary. `GetCurrentAddress()` reports the node the client currently talks to.
-- **The client picks partitions.** The broker never routes: balanced and message-key partitioning are resolved client-side (the message-key hash matches the Rust SDK byte for byte), and consumer-group polls round-robin over the partitions the coordinator assigned to this client.
-- **Consumer groups are assignment-based.** `JoinConsumerGroupAsync` makes this client a member; the assignment is synced on demand and refreshed on every `PingAsync`. The TCP client pings on its own every `HeartbeatInterval` (5 seconds by default), so an idle session survives the server's heartbeat verification and assignments stay fresh.
+- **The client picks partitions.** Balanced and message-key partitioning are resolved client-side (the message-key hash matches the Rust SDK byte for byte), and consumer-group polls round-robin over the partitions the coordinator assigned to this client.
+- **Consumer groups are assignment-based.** `JoinConsumerGroupAsync` makes this client a member. The assignment is synced on demand and refreshed on every `PingAsync`. The TCP client pings on its own every `HeartbeatInterval` (5 seconds by default), so an idle session survives the server's heartbeat verification and assignments stay fresh.
- **Credentials are bounds-checked locally.** A username outside 3-50 bytes, a password outside 3-100 bytes, or a personal access token outside 1-255 bytes is rejected before the request is framed.
-- **Consumer offsets need an explicit partition.** `StoreOffsetAsync` / `DeleteOffsetAsync` do not accept a `null` partition id under VSR; passing one throws client-side.
-- **Polling a topic that does not exist returns an empty poll** rather than throwing. The server answers an unresolved topic with the empty-poll reply shape, so the client cannot tell it apart from a topic with no messages; check the topic exists first if the distinction matters.
+- **Consumer offsets need an explicit partition.** `StoreOffsetAsync` / `DeleteOffsetAsync` do not accept a `null` partition id under VSR. Passing one throws client-side.
+- **Polling a topic that does not exist returns an empty poll** rather than throwing. The server answers an unresolved topic with the empty-poll reply shape, so the client cannot tell it apart from a topic with no messages. Check the topic exists first if the distinction matters.
### Failed requests
The SDK replays a request whenever the server says it never admitted it. Two exceptions surface to the caller:
- `IggyInvalidStatusCodeException` carries the server status code, with `FromServer` telling apart a verdict the cluster reported from a failure the client raised itself.
-- `VsrRequestOutcomeUnknownException` means no server verdict arrived after the request was written (the connection was lost, the call was cancelled, or the server evicted the session mid-flight), so the cluster may or may not have committed it. The SDK will not replay it on a new session, because that would bypass server-side deduplication; re-issuing it is **the caller's decision**.
+- `VsrRequestOutcomeUnknownException` means no server verdict arrived after the request was written (the connection was lost, the call was cancelled, or the server evicted the session mid-flight), so the cluster may or may not have committed it. The SDK will not replay it on a new session, because that would bypass server-side deduplication. Re-issuing it is **the caller's decision**.
The high-level abstractions handle the unknown-outcome case differently:
diff --git a/content/docs/sdk/go/examples.mdx b/content/docs/sdk/go/examples.mdx
index 5e4d4f4c..d8e40011 100644
--- a/content/docs/sdk/go/examples.mdx
+++ b/content/docs/sdk/go/examples.mdx
@@ -16,7 +16,7 @@ If an example fails to log in, the server data directory (`local_data` by defaul
## Getting started
-The [producer](https://github.com/apache/iggy/blob/master/examples/go/getting-started/producer/main.go) creates `sample-stream` and `sample-topic` (if missing) and sends 5 batches of 10 messages to partition `0`; the [consumer](https://github.com/apache/iggy/blob/master/examples/go/getting-started/consumer/main.go) polls them from offset `0` and exits after 5 batches. Note that stream, topic, and partition IDs are **0-based**.
+The [producer](https://github.com/apache/iggy/blob/master/examples/go/getting-started/producer/main.go) creates `sample-stream` and `sample-topic` (if missing) and sends 5 batches of 10 messages to partition `0`. The [consumer](https://github.com/apache/iggy/blob/master/examples/go/getting-started/consumer/main.go) polls them from offset `0` and exits after 5 batches. Stream, topic, and partition IDs are **0-based**.
Run both from the `examples/go` directory:
@@ -25,7 +25,7 @@ go run ./getting-started/producer/main.go
go run ./getting-started/consumer/main.go
```
-You can run multiple producers and consumers simultaneously to observe how messages are distributed across clients.
+Run multiple producers and consumers at once to see how messages are distributed across clients.
Both binaries accept the same flags:
diff --git a/content/docs/sdk/go/intro.mdx b/content/docs/sdk/go/intro.mdx
index 68eec758..04b89268 100644
--- a/content/docs/sdk/go/intro.mdx
+++ b/content/docs/sdk/go/intro.mdx
@@ -6,7 +6,7 @@ The Iggy Go SDK is a client library for interacting with the Iggy server from Go
## Server compatibility
-The SDK and the server must speak the same wire protocol. Tagged releases up to `v0.8.0` predate VSR (and the current API) and pair only with servers of the same era; the module on `master` speaks VSR only and requires a current server. When working from source, build the SDK and the server from the same repository checkout. The code on this page tracks `master`.
+The SDK and the server must speak the same wire protocol. Tagged releases up to `v0.8.0` predate VSR (and the current API) and pair only with servers of the same era. The module on `master` speaks VSR only and requires a current server. When working from source, build the SDK and the server from the same repository checkout. The code on this page tracks `master`.
## Installation
@@ -22,7 +22,7 @@ go get github.com/apache/iggy/foreign/go@master
## Quick start
-The snippets below follow the [getting-started example](https://github.com/apache/iggy/tree/master/examples/go/getting-started). Note that stream, topic, and partition IDs are **0-based**: the first partition of a topic is partition `0`.
+The snippets below follow the [getting-started example](https://github.com/apache/iggy/tree/master/examples/go/getting-started). Stream, topic, and partition IDs are **0-based**: the first partition of a topic is partition `0`.
### Producer
@@ -109,7 +109,7 @@ func main() {
}
```
-`CreateTopic` accepts optional trailing `iggcon.HeaderEntry` values carrying named topic options, for option keys the server catalog gained after the SDK build shipped; `DescribeOptions` lists the keys a server accepts.
+`CreateTopic` accepts optional trailing `iggcon.HeaderEntry` values carrying named topic options, for option keys the server catalog gained after the SDK build shipped. `DescribeOptions` lists the keys a server accepts.
### Consumer
@@ -191,7 +191,7 @@ func main() {
}
```
-Polled message payloads and user headers alias the reply buffer, so retaining one message pins the whole reply; copy the bytes out when they outlive the poll.
+Polled message payloads and user headers alias the reply buffer, so retaining one message pins the whole reply. Copy the bytes out when they outlive the poll.
## TLS
@@ -213,11 +213,11 @@ cli, err := client.NewIggyClient(
## Delivery semantics
-`SendMessages` returns the placements the server committed; delivery is at-least-once. A send whose reply is lost to a dropped connection returns `ErrDisconnected` without a replay: a reconnect registers a fresh client identity, so the server could not deduplicate the replay against a batch that may have already committed. Retrying such a send is the caller's decision and may write the batch twice; consumers that need exactly-once handling deduplicate on the message id. A confirmation re [...]
+`SendMessages` returns the placements the server committed. Delivery is at-least-once. A send whose reply is lost to a dropped connection returns `ErrDisconnected` without a replay: a reconnect registers a fresh client identity, so the server could not deduplicate the replay against a batch that may have already committed. Retrying such a send is the caller's decision and may write the batch twice. Consumers that need exactly-once handling deduplicate on the message id. A confirmation re [...]
## Consumer groups
-The client exposes the full consumer group API: `CreateConsumerGroup`, `JoinConsumerGroup`, `LeaveConsumerGroup`, `GetConsumerGroups`, and `SyncConsumerGroup` (which returns the partitions this client currently owns together with the group generation). To poll as a group member, build the consumer with `iggcon.NewGroupConsumer` and pass a `nil` partition id to `PollMessages`; the client orchestrates the group poll. An empty batch with no assigned partition means nothing to read right now [...]
+The client exposes the full consumer group API: `CreateConsumerGroup`, `JoinConsumerGroup`, `LeaveConsumerGroup`, `GetConsumerGroups`, and `SyncConsumerGroup` (which returns the partitions this client currently owns together with the group generation). To poll as a group member, build the consumer with `iggcon.NewGroupConsumer` and pass a `nil` partition id to `PollMessages`. The client orchestrates the group poll. An empty batch with no assigned partition means nothing to read right now [...]
## Examples
diff --git a/content/docs/sdk/introduction.mdx b/content/docs/sdk/introduction.mdx
index 65c532ae..b2e6c1be 100644
--- a/content/docs/sdk/introduction.mdx
+++ b/content/docs/sdk/introduction.mdx
@@ -21,7 +21,7 @@ The Rust SDK is always the first to receive new features.
## Prerequisites
-The samples across the SDK docs log in with the `iggy`/`iggy` root credentials. These **only exist** when the server is started with the default-credentials flag (or `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` set) on its first boot; otherwise root gets a generated password and the samples fail with `InvalidCredentials`:
+The samples across the SDK docs log in with the `iggy`/`iggy` root credentials. These **only exist** when the server is started with the default-credentials flag (or `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` set) on its first boot. Otherwise root gets a generated password and the samples fail with `InvalidCredentials`:
```bash
cargo run --bin iggy-server -- --fresh --with-default-root-credentials
@@ -114,3 +114,5 @@ When sending messages, you can control partition routing:
| `partition_id(n)` | Send to a specific partition |
| `balanced()` | Round-robin across partitions |
| `messages_key(key)` | Hash-based routing by key |
+
+Strategies resolve in the client on binary transports, and on the server for the HTTP API and the Node SDK.
diff --git a/content/docs/sdk/java/examples.mdx b/content/docs/sdk/java/examples.mdx
index b17ba8ef..ee9accaf 100644
--- a/content/docs/sdk/java/examples.mdx
+++ b/content/docs/sdk/java/examples.mdx
@@ -2,7 +2,7 @@
title: Examples
---
-Runnable examples live in the [examples/java](https://github.com/apache/iggy/tree/master/examples/java) directory of the core repository as a standalone Gradle project. The project builds against the in-repo SDK (via an `includeBuild` substitution), so the examples always match the SDK source in the same checkout. Java 17 is recommended; the included `gradlew` wrapper downloads the pinned Gradle version on first run.
+Runnable examples live in the [examples/java](https://github.com/apache/iggy/tree/master/examples/java) directory of the core repository as a standalone Gradle project. The project builds against the in-repo SDK (via an `includeBuild` substitution), so the examples always match the SDK source in the same checkout. Java 17 is recommended. The included `gradlew` wrapper downloads the pinned Gradle version on first run.
## Starting the server
@@ -81,4 +81,4 @@ Run each example from the `examples/java` directory with its Gradle task:
## Blocking vs. async
-Use the blocking client for scripts, CLI tools, integration tests, and anywhere sequential code is easier to reason about. Use the async client for high throughput, reactive applications, and composing non-blocking requests with `CompletableFuture`. The async client runs I/O on Netty event loop threads: never block them with `.join()`, `.get()`, `Thread.sleep()`, or blocking I/O inside `thenApply`/`thenAccept`; offload blocking work with `thenApplyAsync(fn, executor)`. The [examples/java [...]
+Use the blocking client for scripts, CLI tools, integration tests, and anywhere sequential code is easier to reason about. Use the async client for high throughput, reactive applications, and composing non-blocking requests with `CompletableFuture`. The async client runs I/O on Netty event loop threads: never block them with `.join()`, `.get()`, `Thread.sleep()`, or blocking I/O inside `thenApply`/`thenAccept`. Offload blocking work with `thenApplyAsync(fn, executor)`. The [examples/java [...]
diff --git a/content/docs/sdk/java/intro.mdx b/content/docs/sdk/java/intro.mdx
index 746c111f..f647118c 100644
--- a/content/docs/sdk/java/intro.mdx
+++ b/content/docs/sdk/java/intro.mdx
@@ -30,7 +30,7 @@ Check [Maven Central](https://central.sonatype.com/artifact/org.apache.iggy/iggy
## Quick start
-The snippets below follow the [getting-started example](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/gettingstarted). Note that stream, topic, and partition IDs are **0-based**: the first partition of a topic is partition `0`.
+The snippets below follow the [getting-started example](https://github.com/apache/iggy/tree/master/examples/java/src/main/java/org/apache/iggy/examples/gettingstarted). Stream, topic, and partition IDs are **0-based**: the first partition of a topic is partition `0`.
### Producer
diff --git a/content/docs/sdk/node/examples.mdx b/content/docs/sdk/node/examples.mdx
index f9b9ab8c..9adf48e9 100644
--- a/content/docs/sdk/node/examples.mdx
+++ b/content/docs/sdk/node/examples.mdx
@@ -2,7 +2,7 @@
title: Examples
---
-Working examples are available in the [examples/node](https://github.com/apache/iggy/tree/master/examples/node) directory, written in TypeScript. They are exercised by CI, so they always match the current SDK:
+Working examples are available in the [examples/node](https://github.com/apache/iggy/tree/master/examples/node) directory, written in TypeScript. CI exercises them, so they always match the current SDK:
- **getting-started** - basic producer and consumer
- **basic** - producer and consumer with utilities
diff --git a/content/docs/sdk/node/intro.mdx b/content/docs/sdk/node/intro.mdx
index 3e973b3d..bca8e7fd 100644
--- a/content/docs/sdk/node/intro.mdx
+++ b/content/docs/sdk/node/intro.mdx
@@ -4,7 +4,7 @@ title: Node.js SDK
The Iggy Node.js SDK is a client library that allows you to interact with the Iggy API from your Node.js and TypeScript applications. It communicates with the Iggy server over TCP or TLS using the binary protocol. The package is available on [npm](https://www.npmjs.com/package/apache-iggy) and the source code can be found on [GitHub](https://github.com/apache/iggy/tree/master/foreign/node).
-The SDK speaks the current (VSR) wire protocol exclusively and does not fall back to older formats, so it requires an Iggy server that speaks it. Keep the client and server versions in step: pair the newest package with the newest server.
+The SDK speaks only the current (VSR) wire protocol and doesn't fall back to older formats. Keep the client and server versions in step: pair the newest package with the newest server.
## Installation
@@ -29,7 +29,7 @@ docker run --rm \
cargo run --bin iggy-server -- --fresh --with-default-root-credentials
```
-The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the root credentials the samples below log in with. Without `--with-default-root-credentials` (or the `IGGY_ROOT_USERNAME` / `IGGY_ROOT_PASSWORD` variables), a first boot generates a random root password instead of `iggy`/`iggy`.
+The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the root credentials the samples log in with. Without `--with-default-root-credentials` (or the `IGGY_ROOT_USERNAME` / `IGGY_ROOT_PASSWORD` variables), a first boot generates a random root password instead of `iggy`/`iggy`.
### Producer
@@ -79,6 +79,8 @@ console.log(`Sent ${messages.length} message(s)`);
await client.destroy();
```
+The Node SDK sends the partitioning choice on the wire, and the server resolves balanced and key routing at admission (the Rust, C#, Java, and Go clients instead resolve them client-side).
+
### Consumer
```typescript
@@ -116,12 +118,12 @@ await client.destroy();
## Configuration
-The client constructor accepts a few knobs worth knowing about:
+The client constructor accepts a few options:
- **reconnect**: automatic reconnection with `{ enabled, interval, maxRetries }`.
-- **heartbeatInterval**: the client pings the server every 5 seconds by default; `0` disables heartbeats. The server evicts connections that stay silent too long, so keep it well below the server's heartbeat timeout.
+- **heartbeatInterval**: the client pings the server every 5 seconds by default. `0` disables heartbeats. The server evicts connections that stay silent too long, so keep it well below the server's heartbeat timeout.
- **maxResponseFrameSize**: response frames larger than this limit (default 64 MiB) are rejected and close the connection. Raise it when polling very large batches.
-- **poolSize**: the client currently supports exactly one pooled connection; `min`/`max` above 1 throw.
+- **poolSize**: the client currently supports exactly one pooled connection. `min`/`max` above 1 throw.
- **TLS**: set `transport: 'TLS'` and pass Node `tls.ConnectionOptions` in `options`. See the [tcp-tls example](https://github.com/apache/iggy/tree/master/examples/node/src/tcp-tls).
## Beyond the basics
diff --git a/content/docs/sdk/php/intro.mdx b/content/docs/sdk/php/intro.mdx
index a9c430a1..3b1e08ee 100644
--- a/content/docs/sdk/php/intro.mdx
+++ b/content/docs/sdk/php/intro.mdx
@@ -2,14 +2,14 @@
title: PHP SDK
---
-The Iggy PHP SDK is a native PHP extension built in Rust with [ext-php-rs](https://github.com/davidcole1340/ext-php-rs). It wraps the Rust SDK and exposes a blocking, synchronous `Iggy\Client`: each call drives the extension's internal Tokio runtime and blocks the calling PHP thread until the operation completes. The source code is available on [GitHub](https://github.com/apache/iggy/tree/master/foreign/php).
+The Iggy PHP SDK is a native PHP extension built in Rust with [ext-php-rs](https://github.com/davidcole1340/ext-php-rs). It wraps the Rust SDK and exposes a synchronous `Iggy\Client`: each call drives the extension's internal Tokio runtime and blocks the calling PHP thread until the operation completes. The source code is available on [GitHub](https://github.com/apache/iggy/tree/master/foreign/php).
-The extension is experimental. It is defined as the `apache/iggy-php` composer package (`php-ext` type); to use it today, build it from the repository source.
+The extension is experimental. It's defined as the `apache/iggy-php` composer package (`php-ext` type). To use it today, build it from the repository source.
## Requirements
- Rust and Cargo
-- PHP 8.3 or newer with `php-config` (non-thread-safe builds only; ZTS is not supported)
+- PHP 8.3 or newer with `php-config` (non-thread-safe builds only, no ZTS)
- [`cargo-php`](https://github.com/davidcole1340/ext-php-rs)
## Build and install
@@ -50,7 +50,7 @@ cargo run --bin iggy-server -- --fresh --with-default-root-credentials
The environment variables make the server reachable through the published port (it binds to `127.0.0.1` inside the container by default) and set the `iggy`/`iggy` root credentials the sample's connection string uses.
-Credentials in the connection string are applied on `connect()`. Alternatively, construct the client with a plain address (`new \Iggy\Client('127.0.0.1:8090')`) and call `loginUser('iggy', 'iggy')` explicitly.
+Credentials in the connection string take effect on `connect()`. Alternatively, construct the client with a plain address (`new \Iggy\Client('127.0.0.1:8090')`) and call `loginUser('iggy', 'iggy')` explicitly.
```php
<?php
@@ -119,22 +119,22 @@ foreach ($consumer->iterMessages() as $message) {
}
```
-`AutoCommit::when()` may queue an offset commit before the callback runs. When callback success must control commits, use `AutoCommit::disabled()` and call `storeOffset()` after the work succeeds, as above.
+`AutoCommit::when()` may queue an offset commit before the callback runs. To gate commits on the callback succeeding, use `AutoCommit::disabled()` and call `storeOffset()` after the work succeeds, as above.
## API notes
- Methods are camelCase and classes live in the `Iggy` namespace, for example `Iggy\Client` and `Iggy\SendMessage`.
- Partition ids are 0-based: for a topic with one partition, use `0`.
-- PHP strings are passed as named identifiers, including strings that contain only digits; PHP integers are passed as numeric identifiers.
-- `PollingStrategy::timestamp()` and `timestampMicros()` expect microseconds since the Unix epoch; use `PollingStrategy::timestampSeconds()` for PHP `time()` values.
+- PHP strings are passed as named identifiers, including strings that contain only digits. PHP integers are passed as numeric identifiers.
+- `PollingStrategy::timestamp()` and `timestampMicros()` expect microseconds since the Unix epoch. Use `PollingStrategy::timestampSeconds()` for PHP `time()` values.
- Large unsigned values that can overflow PHP integers, such as message checksums, are returned as decimal strings.
- Errors are thrown as exceptions under `Iggy\Exception`: `AuthenticationException`, `ConnectionException`, `NotFoundException`, `TransientException`, and the `IggyException` base.
- TLS uses the Rust SDK connection-string format, for example `iggy+tcp://user:pass@host:port?tls=true&tls_domain=localhost&tls_ca_file=/path/to/ca.pem`.
-- The extension owns a lazy global Tokio runtime. Do not call `pcntl_fork()` after the first Iggy call: the child inherits file descriptors but not the runtime's worker threads.
+- The extension owns a lazy global Tokio runtime. Don't call `pcntl_fork()` after the first Iggy call: the child inherits file descriptors but not the runtime's worker threads.
## Current status
-`Iggy\Client` currently exposes stream and topic create/get/delete, message sending and polling, consumer groups with offset management, `ping()`, `loginUser()`, and raw binary requests (`sendBinaryRequest()`). Other administration surfaces (users, permissions, personal access tokens, partitions) are not exposed yet.
+`Iggy\Client` currently exposes stream and topic create/get/delete, message sending and polling, consumer groups with offset management, `ping()`, `loginUser()`, and raw binary requests (`sendBinaryRequest()`). Other administration surfaces (users, permissions, personal access tokens, partitions) aren't exposed yet.
## Examples
diff --git a/content/docs/sdk/python/examples.mdx b/content/docs/sdk/python/examples.mdx
index 740492b7..02afce36 100644
--- a/content/docs/sdk/python/examples.mdx
+++ b/content/docs/sdk/python/examples.mdx
@@ -2,7 +2,7 @@
title: Examples
---
-Working examples are available in the [examples/python](https://github.com/apache/iggy/tree/master/examples/python) directory. They are exercised by CI, so they always match the current SDK:
+Working examples are available in the [examples/python](https://github.com/apache/iggy/tree/master/examples/python) directory. CI exercises them, so they always match the current SDK:
- **getting-started** - producer and consumer with optional TLS via `TcpConfig`
- **basic** - producer and consumer using connection strings
diff --git a/content/docs/sdk/python/intro.mdx b/content/docs/sdk/python/intro.mdx
index ec150ac0..8913702e 100644
--- a/content/docs/sdk/python/intro.mdx
+++ b/content/docs/sdk/python/intro.mdx
@@ -4,7 +4,7 @@ title: Python SDK
The Iggy Python SDK is a client library that allows you to interact with the Iggy API from your Python application. It is built as a PyO3 wrapper around the Rust SDK, which means it supports TCP, QUIC, HTTP, and WebSocket transports via connection strings. The package is available on [PyPI](https://pypi.org/project/apache-iggy/) and the source code can be found on [GitHub](https://github.com/apache/iggy/tree/master/foreign/python).
-Because the wheel bundles the Rust SDK, it speaks the current Iggy wire protocol only and does not fall back to older formats. Keep the client and server versions in step: pair the newest wheel with the newest server.
+Because the wheel bundles the Rust SDK, it speaks only the current Iggy wire protocol and doesn't fall back to older formats. Keep the client and server versions in step: pair the newest wheel with the newest server.
## Installation
@@ -114,13 +114,11 @@ asyncio.run(main())
## Beyond the basics
-The SDK covers much more than the quick start above:
-
- **TLS**: construct the client from a `TcpConfig` instead of a connection string and set `tls_enabled` plus `tls_ca_file`. The [getting-started example](https://github.com/apache/iggy/tree/master/examples/python/getting-started) shows the full setup.
- **Consumer groups**: `client.consumer_group(...)` returns an `IggyConsumer` that creates and joins the group by default and commits offsets according to the configured `AutoCommit` mode.
-- **User headers**: `SendMessage(data, user_headers=...)` accepts a plain `dict` with `str`, `bytes`, `bool`, `int`, or `float` values; `HeaderKey`/`HeaderValue` give explicit control over the wire type. See the [message-headers examples](https://github.com/apache/iggy/tree/master/examples/python/message-headers).
-- **Topic options**: `create_topic(..., options=...)` accepts extra option keys as a `dict[str, str]`, validated against the server's catalog; `client.describe_options("topic")` lists the keys a server accepts.
-- **Administration**: user and permission management. Personal access tokens can be used for login via `AutoLogin.personal_access_token(...)` in the connection config; PAT management is not exposed yet.
+- **User headers**: `SendMessage(data, user_headers=...)` accepts a plain `dict` with `str`, `bytes`, `bool`, `int`, or `float` values. `HeaderKey`/`HeaderValue` give explicit control over the wire type. See the [message-headers examples](https://github.com/apache/iggy/tree/master/examples/python/message-headers).
+- **Topic options**: `create_topic(..., options=...)` accepts extra option keys as a `dict[str, str]`, validated against the server's catalog. `client.describe_options("topic")` lists the keys a server accepts.
+- **Administration**: user and permission management. Personal access tokens can be used for login via `AutoLogin.personal_access_token(...)` in the connection config. PAT management isn't exposed yet.
## Examples
diff --git a/content/docs/sdk/rust/examples.mdx b/content/docs/sdk/rust/examples.mdx
index 2201c3ab..71e0e1df 100644
--- a/content/docs/sdk/rust/examples.mdx
+++ b/content/docs/sdk/rust/examples.mdx
@@ -4,13 +4,13 @@ title: Examples
In the core repository, you can find the following [examples](https://github.com/apache/iggy/tree/master/examples/rust/src) using the Rust SDK.
-Before running any of them, start the server with the default root credentials; without the flag (or `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD`) root gets a **generated password** and every example fails with `InvalidCredentials`:
+Before running any of them, start the server with the default root credentials. Without the flag (or `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD`) root gets a **generated password** and every example fails with `InvalidCredentials`:
```bash
cargo run --bin iggy-server -- --fresh --with-default-root-credentials
```
-The examples are cargo example targets of the `iggy_examples` crate. Run them from the `examples/rust` directory with `cargo run --example <name>`. Note that partition IDs are **0-based**: the examples send to and poll from partition `0`.
+The examples are cargo example targets of the `iggy_examples` crate. Run them from the `examples/rust` directory with `cargo run --example <name>`. Partition IDs are **0-based**: the examples send to and poll from partition `0`.
- **[Getting started](https://github.com/apache/iggy/tree/master/examples/rust/src/getting-started)** - the basic example which is discussed in the [getting started](/docs/introduction/getting-started) guide.
@@ -61,7 +61,7 @@ The examples are cargo example targets of the `iggy_examples` crate. Run them fr
cargo run --example typed-headers-consumer
```
-- **[Multi-tenant](https://github.com/apache/iggy/tree/master/examples/rust/src/multi-tenant)** - the comprehensive example of how to structure your application to support multiple tenants (separated by the unique streams) with the different users, permissions, topics etc.
+- **[Multi-tenant](https://github.com/apache/iggy/tree/master/examples/rust/src/multi-tenant)** - the larger example of how to structure your application to support multiple tenants (separated by the unique streams) with the different users, permissions, topics etc.
```bash
cargo run --example multi-tenant-producer
diff --git a/content/docs/sdk/rust/high-level-sdk.mdx b/content/docs/sdk/rust/high-level-sdk.mdx
index fa85bb37..5ddce3e2 100644
--- a/content/docs/sdk/rust/high-level-sdk.mdx
+++ b/content/docs/sdk/rust/high-level-sdk.mdx
@@ -29,17 +29,17 @@ Which can be used to create the client like this:
let client = IggyClient::from_connection_string("iggy://iggy:iggy@localhost:8090")?;
```
-Options are appended as query parameters, for example for TCP:
+Options are appended as query parameters. For TCP:
```bash
iggy://iggy:iggy@localhost:8090?tls=true&tls_domain=test.com&reconnection_retries=5&reconnection_interval=5s&reestablish_after=10s&heartbeat_interval=3s&nodelay=true
```
-The option keys differ per transport (for example QUIC uses `reconnection_max_retries` and `reconnection_reestablish_after` where TCP uses `reconnection_retries` and `reestablish_after`), and unknown keys are rejected as hard errors. See [Connection Strings](/docs/sdk/connection-strings) for the full per-transport option tables and defaults.
+The option keys differ per transport (for example QUIC uses `reconnection_max_retries` and `reconnection_reestablish_after` where TCP uses `reconnection_retries` and `reestablish_after`). Unknown keys are rejected as hard errors. See [Connection Strings](/docs/sdk/connection-strings) for the full per-transport option tables and defaults.
Unless you need to provide a specific implementation of the client-side `Encryptor`, `Partitioner` or adjust some other settings, you should be good to go with the connection string. On the other hand, you can always make use of `IggyClientBuilder::from_connection_string()` to extend the options on top of the provided connection string.
-Additionally, the `IggyClient` internally wraps its transport in an `IggyRwLock`, so a single instance is **cheap to clone** and can be shared across multiple tasks. Typically, it's a good idea to create a separate connection for producing and consuming, but it's not a requirement.
+The `IggyClient` internally wraps its transport in an `IggyRwLock`, so a single instance is **cheap to clone** and can be shared across multiple tasks. Typically, it's a good idea to create a separate connection for producing and consuming, but it's not a requirement.
## Producer
@@ -88,9 +88,9 @@ The `background` mode buffers messages and sends them from shard workers, tradin
| `max_in_flight` | `1` | Maximum batches being sent concurrently. More than 1 can reorder messages when retries occur; `0` means unlimited |
| `error_callback` | `LogErrorCallback` | Async callback invoked on errors the producer cannot recover from |
-Note that `batch_size` limits bytes while `batch_length` limits the message count; whichever limit fires first (including `linger_time`) triggers the flush.
+`batch_size` limits bytes while `batch_length` limits the message count. Whichever limit fires first (including `linger_time`) triggers the flush.
-Call `producer.shutdown().await` to flush the background buffer before exiting; dropping the producer without it **silently discards unflushed messages**.
+Call `producer.shutdown().await` to flush the background buffer before exiting. Dropping the producer without it **silently discards unflushed messages**.
### Send confirmations
@@ -169,11 +169,11 @@ Defaults and lesser-known knobs on `IggyConsumerBuilder`:
`AutoCommitWhen` triggers fire while receiving: `PollingMessages`, `ConsumingAllMessages`, `ConsumingEachMessage`, `ConsumingEveryNthMessage(n)`. `AutoCommitAfter` triggers fire after processing: `ConsumingAllMessages`, `ConsumingEachMessage`, `ConsumingEveryNthMessage(n)`.
-The `After` and `IntervalOrAfter` variants *only* work with `consume_messages()` from the `IggyConsumerMessageExt` trait (see [Stream Builder](/docs/sdk/rust/stream-builder)); the plain `next()` loop cannot observe when your processing finished.
+The `After` and `IntervalOrAfter` variants *only* work with `consume_messages()` from the `IggyConsumerMessageExt` trait (see [Stream Builder](/docs/sdk/rust/stream-builder)). The plain `next()` loop cannot observe when your processing finished.
## Raw requests
For custom commands the `IggyClient` exposes two escape hatches below the typed API:
-- `send_binary_request(code, payload)` sends a raw binary command and returns the raw response. Binary transports only; on HTTP it returns `FeatureUnavailable`. Login and logout codes are rejected with `InvalidCommand`, use `login_user`/`logout_user` so the SDK session state stays correct. Custom codes are forwarded to the server, which decides whether it implements them.
-- `send_http_request(method, path, body)` invokes an arbitrary HTTP endpoint and returns the raw response body. HTTP transport only; binary transports return `FeatureUnavailable`.
+- `send_binary_request(code, payload)` sends a raw binary command and returns the raw response. Binary transports only. On HTTP it returns `FeatureUnavailable`. Login and logout codes are rejected with `InvalidCommand`, use `login_user`/`logout_user` so the SDK session state stays correct. Custom codes are forwarded to the server, which decides whether it implements them.
+- `send_http_request(method, path, body)` invokes an arbitrary HTTP endpoint and returns the raw response body. HTTP transport only. Binary transports return `FeatureUnavailable`.
diff --git a/content/docs/sdk/rust/intro.mdx b/content/docs/sdk/rust/intro.mdx
index 3e59864a..c3fa32a9 100644
--- a/content/docs/sdk/rust/intro.mdx
+++ b/content/docs/sdk/rust/intro.mdx
@@ -63,7 +63,7 @@ iggy://iggy:iggy@my-server:8090?tls=true&tls_ca_file=/path/to/ca.crt
iggy://iggy:iggy@localhost:8090?reconnection_retries=unlimited&heartbeat_interval=5s
```
-Personal Access Tokens are also supported. Any credential without a colon is treated as a PAT; paste the token exactly as the server returned it, with **no prefix**:
+Personal Access Tokens are also supported. Any credential without a colon is treated as a PAT. Paste the token exactly as the server returned it, with **no prefix**:
```
iggy://<personal-access-token>@localhost:8090
@@ -137,6 +137,6 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
}
```
-Note that partition IDs are **0-based**: the first partition of a topic is partition `0`.
+Partition IDs are **0-based**: the first partition of a topic is partition `0`.
For the full getting started tutorial, see [Getting Started](/docs/introduction/getting-started). For the high-level producer/consumer builders, see [High-level SDK](/docs/sdk/rust/high-level-sdk) and [Stream Builder](/docs/sdk/rust/stream-builder).
diff --git a/content/docs/sdk/rust/stream-builder.mdx b/content/docs/sdk/rust/stream-builder.mdx
index 37a28fe6..37eed459 100644
--- a/content/docs/sdk/rust/stream-builder.mdx
+++ b/content/docs/sdk/rust/stream-builder.mdx
@@ -22,7 +22,7 @@ The examples below connect with the `iggy`/`iggy` root credentials, which **only
cargo run --bin iggy-server -- --fresh --with-default-root-credentials
```
-(or with `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` set). Otherwise root gets a generated password and the examples fail with `InvalidCredentials`.
+Setting `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` works too. Otherwise root gets a generated password and the examples fail with `InvalidCredentials`.
## IggyStream Builder
@@ -70,9 +70,9 @@ Here, we use the default configuration to get started quickly, but you can alway
(`IggyStreamConfig::from_stream_topic` or the two per-side configs) to fit your requirements. If you already have a
connected client, use `IggyStream::build(&client, &stream_config)` instead, which returns just the producer and consumer.
-Note that the consumer is bound as `mut`: `consume_messages` (and the underlying message stream) requires **mutable access**.
+The consumer is bound as `mut`: `consume_messages` (and the underlying message stream) requires **mutable access**.
-The `consume_messages` method is not part of the IggyConsumer; it comes from the `IggyConsumerMessageExt`
+The `consume_messages` method is not part of the IggyConsumer. It comes from the `IggyConsumerMessageExt`
trait, which is re-exported by `iggy::prelude`. The PrintEventConsumer implements the MessageConsumer trait
and is invoked to process each incoming message. A simple implementation is shown below:
@@ -203,10 +203,13 @@ async fn main() -> Result<(), IggyError> {
// `Partitioning` specifies to which partition the messages
// should be sent.
// It has the following options:
- // - `Balanced` - round-robin dispatch.
+ // - `Balanced` - the client picks the next partition with a
+ // local round-robin cursor.
// - `PartitionId` - the partition ID is provided by the client.
- // - `MessagesKey` - the partition ID is calculated by the server
- // using the hash of the provided messages key.
+ // - `MessagesKey` - the client hashes the provided messages key
+ // to a partition ID.
+ // The partition count is cached per topic and not refreshed
+ // while the client stays connected.
.partitioning(Partitioning::balanced())
// Sets the retry policy (maximum number of retries and interval)
// in case of messages sending failure.
diff --git a/content/docs/server/benchmarking.mdx b/content/docs/server/benchmarking.mdx
index 780662c2..e1325341 100644
--- a/content/docs/server/benchmarking.mdx
+++ b/content/docs/server/benchmarking.mdx
@@ -8,7 +8,7 @@ We've also built the **[benchmarking platform](https://benchmarks.iggy.apache.or
-Iggy comes with a built-in benchmarking tool `iggy-bench` that measures the performance of the server. It is written in Rust and uses the `tokio` runtime for asynchronous I/O, mimicking the example client applications, so you can use it to estimate the performance of the server in your environment. It is part of the [core repository](https://github.com/apache/iggy/tree/master/core/bench) and lives in the `core/bench` directory.
+Iggy comes with a built-in benchmarking tool, `iggy-bench`. It's written in Rust and uses the `tokio` runtime for asynchronous I/O, mimicking the example client applications, so you can use it to estimate the performance of the server in your environment. It is part of the [core repository](https://github.com/apache/iggy/tree/master/core/bench) and lives in the `core/bench` directory.
@@ -20,7 +20,7 @@ First build the project in release mode:
cargo build --release
```
-**`iggy-bench` does not start a server.** It only connects, and fails with a connection error when nothing is listening. Start `iggy-server` yourself first:
+**`iggy-bench` doesn't start a server.** It only connects, and fails with a connection error when nothing is listening. Start `iggy-server` yourself first:
```bash
cargo run --bin iggy-server -r
@@ -36,7 +36,7 @@ With a server running, invoke the benchmark kind and transport of your choice. T
## Benchmark kinds
-Eight kinds are available; each has a short alias:
+Eight kinds are available, each with a short alias:
1. Pinned producer (`pp`): N producers, each sending to its own stream-topic with a single partition:
@@ -104,8 +104,8 @@ Each transport subcommand accepts a trailing `output` subcommand that persists t
cargo r --bin iggy-bench -r -- pinned-producer tcp output -o performance_results --identifier my-host
```
-`-o/--output-dir` defaults to `performance_results`. Optional flags (`--remark`, `--gitref`, extra info) annotate the run. Persisted results can be inspected with the in-repo report and runner crates (`core/bench/report`, `core/bench/runner`) and browsed or compared in the dashboard (`core/bench/dashboard`), which also powers the public [benchmarking platform](https://benchmarks.iggy.apache.org). A prebuilt dashboard image is available: `docker pull apache/iggy-bench-dashboard`.
+`-o/--output-dir` defaults to `performance_results`. Optional flags (`--remark`, `--gitref`, extra info) annotate the run. You can inspect persisted results with the in-repo report and runner crates (`core/bench/report`, `core/bench/runner`), and browse or compare them in the dashboard (`core/bench/dashboard`), which also powers the public [benchmarking platform](https://benchmarks.iggy.apache.org). A prebuilt dashboard image is available: `docker pull apache/iggy-bench-dashboard`.
## Performance
-The server is thread-per-core and shared-nothing, built on `io_uring` (via `compio`), with shard and CPU pinning configurable under `[system.sharding]`. Throughput and latency depend heavily on hardware, transport, and payload shape (`messages-per-batch * message-size`), so rather than quoting numbers here, run the benchmarks on your own hardware or browse current, dated results on the [benchmarking platform](https://benchmarks.iggy.apache.org).
+The server is thread-per-core and shared-nothing, built on `io_uring` (via `compio`), with shard and CPU pinning configurable under `[system.sharding]`. Throughput and latency depend heavily on hardware, transport, and payload shape (`messages-per-batch * message-size`). Run the benchmarks on your own hardware, or browse current, dated results on the [benchmarking platform](https://benchmarks.iggy.apache.org).
diff --git a/content/docs/server/configuration.mdx b/content/docs/server/configuration.mdx
index 98b52b94..94a95bf0 100644
--- a/content/docs/server/configuration.mdx
+++ b/content/docs/server/configuration.mdx
@@ -2,9 +2,9 @@
title: Configuration
---
-The server reads a single TOML file. A copy of [`core/server/config.toml`](https://github.com/apache/iggy/blob/master/core/server/config.toml) is embedded into the binary and serves as the default configuration, so the server boots with working defaults even without any file on disk. The embedded file is richly commented and is the authoritative reference for the exact version you are running.
+The server reads a single TOML file. A copy of [`core/server/config.toml`](https://github.com/apache/iggy/blob/master/core/server/config.toml) is embedded into the binary and serves as the default configuration, so the server boots with working defaults even without any file on disk. The embedded file is commented throughout and is the authoritative reference for the exact version you're running.
-A config file does not have to be complete. It is **merged over the embedded defaults**, so a minimal file overrides only what it names:
+A config file doesn't have to be complete. It is **merged over the embedded defaults**, so a minimal file overrides only what it names:
```toml
[system]
@@ -22,7 +22,7 @@ address = "0.0.0.0:3000"
Configuration is resolved in three layers. Later layers win:
1. **Embedded defaults**: the `config.toml` compiled into the binary.
-2. **Config file**: the path in `IGGY_CONFIG_PATH`, or `core/server/config.toml` resolved against the current working directory. A missing file is only a warning; the server continues on embedded defaults.
+2. **Config file**: the path in `IGGY_CONFIG_PATH`, or `core/server/config.toml` resolved against the current working directory. A missing file is only a warning. The server continues on embedded defaults.
3. **Environment variables**: any `IGGY_`-prefixed override.
Before the environment is read, the server loads a `.env` file from the working directory, or from the path named by `IGGY_ENV_PATH`.
@@ -41,7 +41,7 @@ IGGY_SYSTEM_LOGGING_LEVEL=debug # [system.logging] level
IGGY_SYSTEM_SHARDING_CPU_ALLOCATION=4 # [system.sharding] cpu_allocation
```
-Two variables live outside the config schema: `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` set the root credentials, always as a pair. **Only the first creation** of the root user reads them; on an existing data directory the stored root user is recovered unchanged.
+Two variables live outside the config schema: `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` set the root credentials, always as a pair. **Only the first creation** of the root user reads them. On an existing data directory the stored root user is recovered unchanged.
### Secrets
@@ -49,8 +49,8 @@ Five values are secret-flagged: `http.jwt.encoding_secret`, `http.jwt.decoding_s
### Validation at boot
-- Unknown keys are **silently ignored**. A typo in a key name means the default silently stays in force, so check spelling when a setting does not seem to take effect. The [relocated keys](#relocated-configuration-keys) below are the exception: they refuse boot.
-- `system.segment.archive_expired = true` and `system.recovery.recreate_missing_state = true` are unsupported placeholders; setting either aborts boot.
+- Unknown keys are **silently ignored**. A typo in a key name leaves the default in force, so check spelling when a setting doesn't seem to take effect. The [relocated keys](#relocated-configuration-keys) below are the exception: they refuse boot.
+- `system.segment.archive_expired = true` and `system.recovery.recreate_missing_state = true` are unsupported placeholders. Setting either aborts boot.
- Several sections validate relationships between keys (QUIC windows, sharding shutdown budgets, metadata journal sizing, partition transfer floors). A violation aborts boot with an error naming the keys. The constraints are listed with their sections below.
- The `RUST_LOG` environment variable **always takes precedence** over `system.logging.level`.
@@ -66,7 +66,7 @@ Five values are secret-flagged: `http.jwt.encoding_secret`, `http.jwt.decoding_s
## Relocated configuration keys
-Retention, durability, and segment layout are no longer server-wide settings. They became per-topic options set at topic creation; see [Topic options](/docs/server/topic-options). The server **refuses to boot** while any of the old keys is still present, in the config file or in the environment, and the error names the replacement. This is deliberate: the storage knobs are create-only options now, so a topic created while an old key was silently ignored could never be given the setting a [...]
+Retention, durability, and segment layout are no longer server-wide settings. They became per-topic options set at topic creation. See [Topic options](/docs/server/topic-options). The server **refuses to boot** while any of the old keys is still present, in the config file or in the environment, and the error names the replacement. This is deliberate: the storage knobs are create-only options now, so a topic created while an old key was silently ignored could never be given the setting a [...]
| Old server key | Replacement |
|----------------|-------------|
@@ -80,11 +80,11 @@ Retention, durability, and segment layout are no longer server-wide settings. Th
| `system.message_deduplication` (whole section) | Removed with the feature. No replacement. |
| `extra` (whole section) | Removed. Admission caps are compile-time constants. No replacement. |
-Other sections from older configs (`[message_saver]`, `[system.state]`, `[system.backup]`, `[system.compression]`, `[tcp.socket]`, segment `cache_indexes`, and similar) were removed without a boot check. Like any unknown key they are silently ignored; remove them from your file.
+Other sections from older configs (`[message_saver]`, `[system.state]`, `[system.backup]`, `[system.compression]`, `[tcp.socket]`, segment `cache_indexes`, and similar) were removed without a boot check. Like any unknown key they're silently ignored. Remove them from your file.
## Reference
-The tables below list every section with its shipped defaults. In cluster mode, every enabled transport takes its **port** from this node's `cluster.nodes` `ports` entry; the transport's own `address` picks only the bind interface.
+The tables below list every section with its shipped defaults. In cluster mode, every enabled transport takes its **port** from this node's `cluster.nodes` `ports` entry. The transport's own `address` picks only the bind interface.
### `[http]`
@@ -95,7 +95,7 @@ The tables below list every section with its shipped defaults. In cluster mode,
| `max_request_size` | `"2 MB"` | Maximum request body size. |
| `web_ui` | `false` | Serve the embedded Web UI at `/ui`. Requires a server built with the `iggy-web` feature; without it, `true` logs a warning and the server continues. |
-HTTP sessions hold live server-side session state (they count against `metadata.clients_table_max`). Consumer group management (create, get, delete) is available over HTTP; group membership (join, leave) is not and needs a stateful transport.
+HTTP sessions hold live server-side session state (they count against `metadata.clients_table_max`). Consumer group management (create, get, delete) is available over HTTP. Group membership (join, leave) is not and needs a stateful transport.
In cluster mode, followers forward control-plane requests (streams, topics, users, and so on) to the current primary when a cluster-wide JWT key exists (see `[http.jwt]` below). Forwarding does not cover the partition plane: message produce and consumer-offset writes must reach the partition's primary node directly, while message polls read locally on any node.
@@ -143,7 +143,7 @@ jwks_url = "https://issuer.example.com/.well-known/jwks.json"
user_id = 1
```
-`issuer`, `audience`, and `jwks_url` are required per entry. `user_id` is the non-zero iggy user every token from that issuer is remapped onto; it defaults to `0` (root), which is **rejected**, so set it explicitly.
+`issuer`, `audience`, and `jwks_url` are required per entry. `user_id` is the non-zero iggy user every token from that issuer is remapped onto. It defaults to `0` (root), which is **rejected**, so set it explicitly.
Enabling an issuer opens an outbound JWKS fetch that is reachable before a token's signature is verified: a token naming the issuer with an unknown key id can trigger a fetch to `jwks_url`. The target is fixed (not attacker-chosen), concurrent misses coalesce onto one fetch, and repeats are rate-limited to one outbound request per issuer per short window. The same window bounds how quickly a freshly rotated signing key is picked up.
@@ -154,7 +154,7 @@ Enabling an issuer opens an outbound JWKS fetch that is reachable before a token
| `enabled` | `true` | Expose Prometheus metrics. |
| `endpoint` | `"/metrics"` | Metrics path. Must start with `/`. |
-The metrics route **requires authentication** like every other read: a missing or invalid bearer credential is rejected with 401. Any authenticated user may scrape; there is no extra RBAC rule. Scrapers present a JWT or a personal access token as the bearer:
+The metrics route **requires authentication** like every other read: a missing or invalid bearer credential is rejected with 401. Any authenticated user may scrape. There's no extra RBAC rule. Scrapers present a JWT or a personal access token as the bearer:
```yaml
scrape_configs:
@@ -175,7 +175,7 @@ scrape_configs:
| `cert_file` | `"core/certs/iggy_cert.pem"` | PEM certificate path. |
| `key_file` | `"core/certs/iggy_key.pem"` | PEM key path. |
-HTTP TLS is **file-based only**; there is no `self_signed` option for this listener. Both files must exist when enabled.
+HTTP TLS is **file-based only**. There's no `self_signed` option for this listener. Both files must exist when enabled.
### `[tcp]`
@@ -245,7 +245,7 @@ The size knobs are optional. Leaving one unset keeps the `compio-ws` (tungstenit
### TLS across transports
-The `self_signed` flag does not mean the same thing everywhere:
+The `self_signed` flag doesn't mean the same thing everywhere:
| Listener | Behavior |
|----------|----------|
@@ -290,7 +290,7 @@ Storage sub-paths chain relative to each other: `[system.runtime] path = "runtim
| `path` | `"partitions"` | Partition data directory, relative to the topic path. |
| `validate_checksum` | `true` | Re-hash every batch a disk poll reads and fail the poll closed on a mismatch, so a segment damaged at rest is reported instead of served. `false` serves whatever decodes, which can hand a consumer bytes provably not the ones written. Only turn it off with a corruption guard elsewhere in the stack. |
-Durability and flush cadence (`enforce_fsync`, `messages_required_to_save`, `size_of_messages_required_to_save`) are per-topic creation options; see [Topic options](/docs/server/topic-options).
+Durability and flush cadence (`enforce_fsync`, `messages_required_to_save`, `size_of_messages_required_to_save`) are per-topic creation options. See [Topic options](/docs/server/topic-options).
### `[system.segment]` and `[system.recovery]`
@@ -428,10 +428,10 @@ Cluster mode is configured here but documented in [Clustering](/docs/clustering/
| `repair_retry_interval` | `"1s"` | Re-request cadence for a stalled journal-repair stream. |
| `repair_chunk_max` | `128` | Prepares served per repair round. Must stay strictly below `message_bus.peer_queue_capacity`. Between 1 and 1024. |
-Durations are **rounded down** to the consensus 10 ms tick; values under 10 ms become one tick.
+Durations are **rounded down** to the consensus 10 ms tick. Values under 10 ms become one tick.
Sub-sections:
-- `[cluster.auth]`: replica-to-replica authentication (PSK plus BLAKE3 keyed-MAC handshake). `shared_secret` must be at least 32 bytes of CSPRNG output, identical on every node; prefer `IGGY_CLUSTER_AUTH_SHARED_SECRET`. `previous_shared_secret` enables rolling key rotation.
-- `[cluster.tls]`: TLS 1.3 on the replica port. Requires `cluster.auth.enabled`; the PSK authenticates the peer, TLS supplies confidentiality.
+- `[cluster.auth]`: replica-to-replica authentication (PSK plus BLAKE3 keyed-MAC handshake). `shared_secret` must be at least 32 bytes of CSPRNG output, identical on every node. Prefer `IGGY_CLUSTER_AUTH_SHARED_SECRET`. `previous_shared_secret` enables rolling key rotation.
+- `[cluster.tls]`: TLS 1.3 on the replica port. Requires `cluster.auth.enabled`. The PSK authenticates the peer, TLS supplies confidentiality.
- `[[cluster.nodes]]`: the full roster, byte-identical on every node. Each entry has `name`, `ip`, `replica_id`, and `ports` (`tcp`, `quic`, `http`, `websocket`, `tcp_replica`). In cluster mode `ports` is the single source of listener ports: every enabled transport needs an explicit per-node port or the server refuses to start. `advertised_address` and per-CIDR `advertised_addresses` selectors control what clients are told to dial.
diff --git a/content/docs/server/docker.mdx b/content/docs/server/docker.mdx
index 703ba5f3..08d07590 100644
--- a/content/docs/server/docker.mdx
+++ b/content/docs/server/docker.mdx
@@ -84,7 +84,7 @@ The repository root carries its own `Dockerfile` and `docker-compose.yml` for bu
docker build -t iggy .
```
-Or use `docker compose up` directly from the repository root. Note that this image **differs from the published one**: it sets the `0.0.0.0` listener addresses via `ENV` and keeps its data at `/local_data`.
+Or use `docker compose up` directly from the repository root. That image **differs from the published one**: it sets the `0.0.0.0` listener addresses via `ENV` and keeps its data at `/local_data`.
### Running the CLI inside the container
@@ -96,7 +96,7 @@ docker exec -it iggy iggy -u iggy -p Secret123 stream list
## Helm charts
-Helm charts for Kubernetes deployment are available in the [repository](https://github.com/apache/iggy/tree/master/helm/charts/iggy). Almost everything server-related nests under the `server:` key in `values.yaml`; the chart also bundles a Web UI deployment under `ui:`.
+Helm charts for Kubernetes deployment are available in the [repository](https://github.com/apache/iggy/tree/master/helm/charts/iggy). Almost everything server-related nests under the `server:` key in `values.yaml`. The chart also bundles a Web UI deployment under `ui:`.
### Quick start
@@ -109,7 +109,7 @@ helm install iggy ./helm/charts/iggy
- **Server Deployment** - runs `apache/iggy` with the pod security context the server needs: seccomp profile `Unconfined` (for `io_uring`) plus the `IPC_LOCK` capability (for memory locking). Listener addresses are set to `0.0.0.0` via `server.env`, and the data volume mounts at `/app/local_data`.
- **Server Service** - exposes the `http` (3000), `quic` (8080), and `tcp` (8090) ports. WebSocket is not exposed by the chart.
- **Web UI Deployment + Service** - a separate `apache/iggy-web-ui` deployment on port 3050, enabled by default (`ui.enabled`).
-- **Secret** - root user credentials from `server.users.root` (default `iggy`/`changeit`); point `existingSecret` at your own Secret in production.
+- **Secret** - root user credentials from `server.users.root` (default `iggy`/`changeit`). Point `existingSecret` at your own Secret in production.
- **PersistentVolumeClaim** - storage for `/app/local_data`, **disabled by default** (`server.persistence.enabled`), 8Gi when enabled.
- **ServiceAccount**, **HPA**, **Ingress** - the usual optional plumbing, for both server and UI.
- **ServiceMonitor** - Prometheus scrape config (optional). The `/metrics` endpoint requires a bearer credential like every other read, so wire a token (e.g. a personal access token stored in a Secret) into `server.serviceMonitor.authorization` - an unauthenticated scrape gets 401.
diff --git a/content/docs/server/introduction.mdx b/content/docs/server/introduction.mdx
index 2fd96a70..38787ec3 100644
--- a/content/docs/server/introduction.mdx
+++ b/content/docs/server/introduction.mdx
@@ -8,11 +8,11 @@ Iggy server is the most important part of the system as it's responsible for han
The releases are published to GitHub and can be found [here](https://github.com/apache/iggy/tags). The official Docker images can be found [here](https://hub.docker.com/r/apache/iggy), simply type `docker pull apache/iggy`.
-Keep in mind, that if you were to compile the source code in release mode, the longer compilation time is due to [LTO](https://doc.rust-lang.org/rustc/linker-plugin-lto.html) enabled in the `[profile.release]` section of the workspace [Cargo.toml](https://github.com/apache/iggy/blob/master/Cargo.toml).
+If you compile the source code in release mode, the longer compilation time comes from [LTO](https://doc.rust-lang.org/rustc/linker-plugin-lto.html) enabled in the `[profile.release]` section of the workspace [Cargo.toml](https://github.com/apache/iggy/blob/master/Cargo.toml).
## Running the server
-One `iggy-server` binary serves both the single-node and the clustered deployment; the loaded configuration decides which one you get. The server accepts three CLI flags:
+One `iggy-server` binary serves both the single-node and the clustered deployment. The loaded configuration decides which one you get. The server accepts three CLI flags:
| Flag | Purpose |
|------|---------|
@@ -20,9 +20,9 @@ One `iggy-server` binary serves both the single-node and the clustered deploymen
| `--with-default-root-credentials` | Set the root credentials to `iggy`/`iggy` on first start, unless `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` are already set. Development only. |
| `--replica-id <N>` | Select this node's entry in `cluster.nodes`. Required when `cluster.enabled = true`. See [Clustering](/docs/clustering/vsr). |
-Configuration comes from the TOML file named by `IGGY_CONFIG_PATH`; without one, the server boots on the defaults embedded in the binary. Any key can be overridden with an `IGGY_`-prefixed environment variable, and a `.env` file in the working directory (or the file named by `IGGY_ENV_PATH`) is loaded at startup. See [Configuration](/docs/server/configuration) for the full reference.
+Configuration comes from the TOML file named by `IGGY_CONFIG_PATH`. Without one, the server boots on the defaults embedded in the binary. Any key can be overridden with an `IGGY_`-prefixed environment variable, and a `.env` file in the working directory (or the file named by `IGGY_ENV_PATH`) is loaded at startup. See [Configuration](/docs/server/configuration) for the full reference.
-When no root credentials are provided on the very first start, the server generates a random root password and prints it to the log. That is the *only* time it can be read.
+When no root credentials are provided on the very first start, the server generates a random root password and prints it to the log. That's the *only* time it can be read.
The HTTP API endpoints can be found in [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http) file, which can be used with [REST Client](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) extension for VS Code.
@@ -35,8 +35,8 @@ cargo run --bin iggy-server -- --fresh --with-default-root-credentials
cargo run --bin data-seeder-tool
```
-The seeder logs in as `iggy`/`iggy` by default; pass `--username` and `--password` to use other credentials.
+The seeder logs in as `iggy`/`iggy` by default. Pass `--username` and `--password` to use other credentials.
## Authentication
-Only the ping liveness probe and the login handshake itself (username/password, personal access token, or HTTP token refresh, all of which prove a credential) are served without an authenticated session. Every other request requires one and is subject to [permissions](/docs/server/security): fetching server stats, for example, needs the `read_servers` permission (the root user has it), and even the Prometheus `/metrics` scrape **must present a bearer credential**. A session is created by [...]
+Only the ping liveness probe and the login handshake itself (username/password, personal access token, or HTTP token refresh, all of which prove a credential) are served without an authenticated session. Every other request requires one and is subject to [permissions](/docs/server/security): fetching server stats, for example, needs the `read_servers` permission (the root user has it), and even the Prometheus `/metrics` scrape **must present a bearer credential**. A session is created by [...]
diff --git a/content/docs/server/networking.mdx b/content/docs/server/networking.mdx
index f14af29d..a136e68e 100644
--- a/content/docs/server/networking.mdx
+++ b/content/docs/server/networking.mdx
@@ -21,7 +21,7 @@ Every request and reply on the stateful transports starts with a fixed 256-byte
## Connection handling across shards
-The server runs one shard per core (thread-per-core, shared-nothing). All listeners - every client transport - bind on shard 0 only. Shard 0's coordinator hands accepted plaintext TCP and WebSocket connections to peer shards round-robin by transferring the file descriptor at accept time; from then on the connection lives entirely on its shard. QUIC, TCP with TLS, and HTTP terminate on shard 0, because their per-connection state cannot move between threads. Sockets **never migrate** after [...]
+The server runs one shard per core (thread-per-core, shared-nothing). All listeners (every client transport) bind on shard 0 only. Shard 0's coordinator hands accepted plaintext TCP and WebSocket connections to peer shards round-robin by transferring the file descriptor at accept time. From then on the connection lives entirely on its shard. QUIC, TCP with TLS, and HTTP terminate on shard 0, because their per-connection state cannot move between threads. Sockets **never migrate** after a [...]
## TCP
@@ -41,7 +41,7 @@ key_file = "core/certs/iggy_key.pem"
## QUIC
-QUIC provides built-in TLS encryption and runs the same binary protocol over UDP, built on the quinn library. While not quite as fast as TCP, QUIC provides better security out of the box (TLS is **mandatory**). Defaults:
+QUIC runs the same binary protocol over UDP, on top of the quinn library, with TLS built in. It isn't quite as fast as TCP, but security comes out of the box (TLS is **mandatory**). Defaults:
```toml
[quic]
@@ -63,9 +63,9 @@ key_file = ""
A few of these are deliberate rather than arbitrary:
-- `max_concurrent_bidi_streams = 1`: the SDK opens a fresh stream per command and the server accepts one at a time, so a connection carries one in-flight command; raising the cap does **not** make handlers concurrent.
-- `initial_mtu = "1200 B"` is the QUIC minimum; MTU discovery probes upward automatically, and boot rejects values below 1200.
-- `stream_receive_window` is the per-stream slice of `receive_window`, equal to it because there is a single stream; lower it if you ever raise the stream cap.
+- `max_concurrent_bidi_streams = 1`: the SDK opens a fresh stream per command and the server accepts one at a time, so a connection carries one in-flight command. Raising the cap does **not** make handlers concurrent.
+- `initial_mtu = "1200 B"` is the QUIC minimum. MTU discovery probes upward automatically, and boot rejects values below 1200.
+- `stream_receive_window` is the per-stream slice of `receive_window`, equal to it because there is a single stream. Lower it if you ever raise the stream cap.
- `keep_alive_interval` is one third of `max_idle_timeout`, so two consecutive lost keep-alives fit before the idle timer closes the connection.
## HTTP
@@ -83,7 +83,7 @@ The HTTP server also hosts:
WebSocket provides bidirectional streaming over HTTP upgrade. Iggy uses its own `compio-ws` implementation that bridges tungstenite's poll-based model with compio's completion-based I/O, reading frames through a buffered stream with a 128 KiB base buffer that can grow to a 64 MiB cap.
-The `[websocket]` section exposes the frame-tuning knobs (`read_buffer_size`, `write_buffer_size`, `max_write_buffer_size`, `max_message_size`, `max_frame_size`, `accept_unmasked_frames`); see [Configuration](/docs/server/configuration). TLS for WebSocket has its own `[websocket.tls]` section.
+The `[websocket]` section exposes the frame-tuning knobs (`read_buffer_size`, `write_buffer_size`, `max_write_buffer_size`, `max_message_size`, `max_frame_size`, `accept_unmasked_frames`). See [Configuration](/docs/server/configuration). TLS for WebSocket has its own `[websocket.tls]` section.
**Benchmark comparison** (AWS i3en.3xlarge, fsync per message, 4 producers, 40M messages):
@@ -101,7 +101,7 @@ Each transport configures TLS in its own section, and the `self_signed` semantic
| QUIC | `[quic.certificate]` | TLS is mandatory (part of the QUIC spec). `self_signed = true` **always** generates an ephemeral certificate and ignores `cert_file`/`key_file`, logging a warning if the files exist. |
| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires real `cert_file`/`key_file`. |
-With `self_signed = false`, `cert_file` and `key_file` must both exist. For production, provide proper certificate files everywhere; ephemeral certificates **change on every start** and cannot be verified by clients.
+With `self_signed = false`, `cert_file` and `key_file` must both exist. For production, provide proper certificate files everywhere. Ephemeral certificates **change on every start** and cannot be verified by clients.
## Heartbeat
@@ -111,7 +111,7 @@ enabled = true
interval = "30 s"
```
-A connection that sends nothing (no request, no ping) for 1.2x the interval has its session released so its consumer groups rebalance off it. Only connections holding a consumer-group membership are evicted; others are left alone until their socket closes.
+A connection that sends nothing (no request, no ping) for 1.2x the interval has its session released so its consumer groups rebalance off it. Only connections holding a consumer-group membership are evicted. Others are left alone until their socket closes.
Most SDKs (Rust, Go, Python, Node, async Java) ping automatically every 5 seconds from a connected high-level client, well inside the resulting 36-second window. The blocking Java and C# SDKs have **no automatic heartbeat**, so an idle consumer-group member there must ping manually or reconnect after eviction.
diff --git a/content/docs/server/schema.mdx b/content/docs/server/schema.mdx
index 1f68fb68..83a3ebac 100644
--- a/content/docs/server/schema.mdx
+++ b/content/docs/server/schema.mdx
@@ -4,7 +4,7 @@ title: Schema
Iggy speaks one binary protocol over its three binary transports: TCP, QUIC, and WebSocket. Commands, responses, data models, and status codes are the same on all of them. The HTTP transport is separate: it exposes the same operations as JSON REST endpoints, listed in [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http).
-This page describes **binary protocol version 0.11.0**. The protocol version is the semver of the `iggy_binary_protocol` crate, and it is exchanged and checked during login (see [Connection lifecycle](#connection-lifecycle)). The crate itself is the source of truth: every request and response module carries a `Wire format:` doc comment, and this page is generated against those.
+This page describes **binary protocol version 0.11.0**. The protocol version is the semver of the `iggy_binary_protocol` crate, and it's exchanged and checked during login (see [Connection lifecycle](#connection-lifecycle)). The crate itself is the source of truth: every request and response module carries a `Wire format:` doc comment, and this page is generated against those.
All multi-byte integers are **little-endian** unless stated otherwise.
@@ -32,7 +32,7 @@ The header shape is identified by the `command` byte at offset 60. Values a clie
Other `command` values (Prepare, PrepareOk, view-change traffic, and so on) are replica-to-replica consensus messages and never appear on a client connection.
-The `size` field at offset 48 is the total frame length in bytes, header included. A body, when present, immediately follows the 256 header bytes and has length `size - 256`. Readers can decode `size` before typing the header; that offset is a protocol constant.
+The `size` field at offset 48 is the total frame length in bytes, header included. A body, when present, immediately follows the 256 header bytes and has length `size - 256`. Readers can decode `size` before typing the header. That offset is a protocol constant.
### RequestHeader
@@ -61,7 +61,7 @@ Client to server. 256 bytes.
### Operation discriminants
-The `operation` byte tells the server which state-machine operation the request carries. Replicated operations are identified by `operation` alone; non-replicated operations (reads, ping) all use `operation = 2` and carry their concrete u32 command code in header bytes 196..200.
+The `operation` byte tells the server which state-machine operation the request carries. Replicated operations are identified by `operation` alone. Non-replicated operations (reads, ping) all use `operation = 2` and carry their concrete u32 command code in header bytes 196..200.
Values a client may send:
@@ -102,14 +102,14 @@ The planes matter for delivery semantics:
- **Metadata operations** replicate through the metadata consensus group. The server deduplicates them by `(client, request)` and caches replies, so a retried request gets the cached answer instead of a double apply (exactly-once).
- **Partition operations** replicate through their partition's consensus group. They are at-least-once: no reply cache, a replay may apply again.
-- **Non-replicated operations** are reads; they bypass consensus and deduplication entirely.
+- **Non-replicated operations** are reads. They bypass consensus and deduplication entirely.
### Request numbering
`request` is a per-client counter the server's client table tracks for metadata-plane operations:
- Metadata operations must send a strictly increasing `request` (the SDK advances the counter per metadata request).
-- Partition operations and non-replicated operations send the current counter value without advancing it; the server does not track theirs.
+- Partition operations and non-replicated operations send the current counter value without advancing it. The server doesn't track theirs.
- The counter is a watermark, not a contiguous sequence: any value above the last accepted one is admissible.
`session` is the fence epoch: the value handed back by the login reply. Every request after login must echo it. When the same `client` id registers again, the new registration mints a higher epoch and requests carrying the old one are fenced (rejected as zombies).
@@ -145,7 +145,7 @@ Decode order for a client:
1. Read 256 bytes, check `command`. `13` (`Eviction`) means the session is dead: map the eviction reason to an error and stop. `8` (`Reply`) continues.
2. Read the remaining `size - 256` body bytes.
-3. If `status` is nonzero, the request failed pre-commit; the status value is the `IggyError` code and the body is empty.
+3. If `status` is nonzero, the request failed pre-commit. The status value is the `IggyError` code and the body is empty.
4. Otherwise decode the body. For result-framed operations, strip the [result section](#result-section) first.
### Result section
@@ -160,9 +160,9 @@ count x { index: u32, result: u32 }
- Success: `count = 0`, and the typed response payload (if any) follows the 4 count bytes.
- Committed business rejection: one entry `{ index: 0, result: error_code }` and no payload.
-The header `status` channel and the result section are mutually exclusive by construction: a reply either failed pre-commit (`status` nonzero, empty body) or committed (`status` zero, result section present). A login-register reply carries the result section only when non-empty; on success its body starts directly with the [login response payload](#login-register).
+The header `status` channel and the result section are mutually exclusive by construction: a reply either failed pre-commit (`status` nonzero, empty body) or committed (`status` zero, result section present). A login-register reply carries the result section only when non-empty. On success its body starts directly with the [login response payload](#login-register).
-Replies to non-replicated commands are not result-framed: after `status = 0` the body is the response payload directly.
+Replies to non-replicated commands aren't result-framed: after `status = 0` the body is the response payload directly.
### EvictionHeader
@@ -235,7 +235,7 @@ Both request bodies begin with the `ClientVersionInfo` prefix, so the server can
[sdk_version_len: u8][sdk_version: UTF-8, 1-255 bytes]
```
-`protocol_version` is the packed version of the protocol the client was built against. `sdk_name` identifies the SDK (for example `rust-sdk`, `go-sdk`); `sdk_version` is the SDK's own build version.
+`protocol_version` is the packed version of the protocol the client was built against. `sdk_name` identifies the SDK (for example `rust-sdk`, `go-sdk`). `sdk_version` is the SDK's own build version.
After the prefix:
@@ -269,11 +269,11 @@ A successful login is a normal `Reply` whose body is:
The client stores `session` and echoes it in every subsequent request header. Logout is command code **39** (`LOGOUT_USER`), `operation = 3` (`Logout`), empty body.
-The legacy login commands **38** (`LOGIN_USER`) and **44** (`LOGIN_WITH_PERSONAL_ACCESS_TOKEN`) are refused: the server answers them with an `Eviction` frame, reason `15` (`MalformedLogin`). Only `PING` is accepted before login; every other command on an unauthenticated connection is denied.
+The legacy login commands **38** (`LOGIN_USER`) and **44** (`LOGIN_WITH_PERSONAL_ACCESS_TOKEN`) are refused: the server answers them with an `Eviction` frame, reason `15` (`MalformedLogin`). Only `PING` is accepted before login. Every other command on an unauthenticated connection is denied.
## Command catalog
-The u32 command codes. Replicated commands are identified on the wire by their `operation` byte (see [Operation discriminants](#operation-discriminants)); the code column below is the protocol-level registry and, for non-replicated commands, the value carried in header bytes 196..200.
+The u32 command codes. Replicated commands are identified on the wire by their `operation` byte (see [Operation discriminants](#operation-discriminants)). The code column below is the protocol-level registry and, for non-replicated commands, the value carried in header bytes 196..200.
```bash
# System
@@ -390,7 +390,7 @@ Several commands take an optional partition id encoded as 5 fixed bytes:
[flag: u8][partition_id: u32]
```
-`flag = 1` means the id is set; `flag = 0` means none (the value bytes are then zero). For consumer-group consumers the partition id is left unset and the server resolves it.
+`flag = 1` means the id is set. `flag = 0` means none (the value bytes are then zero). For consumer-group consumers the partition id is left unset and the server resolves it.
### Partitioning
@@ -400,10 +400,12 @@ How `SendMessages` picks the target partition:
[kind: u8][length: u8][value: 0-255 bytes]
```
-- `Balanced`: `kind = 1`, `length = 0`, no value. Server balances across partitions.
+- `Balanced`: `kind = 1`, `length = 0`, no value. The server resolves the partition at admission with a per-node round-robin counter.
- `PartitionId`: `kind = 2`, `length = 4`, value is a u32 partition id.
- `MessagesKey`: `kind = 3`, `length = 1..255`, value is a routing key hashed to a partition.
+First-party binary SDKs pre-resolve balanced and key routing client-side and normally send `kind = 2`.
+
### Polling strategy
Fixed 9 bytes:
@@ -434,7 +436,7 @@ Typed values used by user headers and option values. The kind byte:
| 6 | Int32 | 14 | Float32 |
| 7 | Int64 | 15 | Float64 |
-Integers are little-endian; `Bool` is one byte.
+Integers are little-endian. `Bool` is one byte.
### User headers block
@@ -448,7 +450,7 @@ Fields pair up: first the key, then the value. Keys are `String` kind. Every `le
### Options block
-`CreateStream`, `UpdateStream`, `CreateTopic`, `UpdateTopic`, `CreateUser`, and `UpdateUser` end with a key-value **options block** that reuses the user-headers TLV encoding. The block runs from its start to the end of the payload (no length prefix; the validator requires exact consumption) and may be empty.
+`CreateStream`, `UpdateStream`, `CreateTopic`, `UpdateTopic`, `CreateUser`, and `UpdateUser` end with a key-value **options block** that reuses the user-headers TLV encoding. The block runs from its start to the end of the payload (no length prefix, and the validator requires exact consumption) and may be empty.
On top of the TLV walk, options enforce: string keys only (kind 2, valid UTF-8), no duplicate keys, at most 1024 entries, and at most 100000 bytes total.
@@ -458,7 +460,7 @@ Semantics:
- **Update** requests are patches: keys absent from the block are left alone, never reset. A client built before a key existed cannot erase it.
- Unknown keys are rejected at the wire edge, never silently skipped.
-The catalog is discoverable at runtime with `DESCRIBE_OPTIONS` (code 13), payload `[scope: u8]` with scope `1` = topic, `2` = stream, `3` = user (HTTP: `GET /options/topic`). Today only topics have keys; the stream and user catalogs are empty and any key sent for them is rejected. The topic catalog (`segment_size`, `enforce_fsync`, `message_expiry`, `max_topic_size`, and the rest) with defaults and constraints is documented on the [Topic options](/docs/server/topic-options) page. `Update [...]
+The catalog is discoverable at runtime with `DESCRIBE_OPTIONS` (code 13), payload `[scope: u8]` with scope `1` = topic, `2` = stream, `3` = user (HTTP: `GET /options/topic`). Today only topics have keys. The stream and user catalogs are empty, and any key sent for them is rejected. The topic catalog (`segment_size`, `enforce_fsync`, `message_expiry`, `max_topic_size`, and the rest) with defaults and constraints is documented on the [Topic options](/docs/server/topic-options) page. `Updat [...]
### Compression
@@ -469,7 +471,7 @@ The `compression_algorithm` value used by topic options:
| 1, `none` | No compression (default) |
| 2, `gzip` | Gzip |
-Other algorithms (lz4, zstd) are planned but not implemented; any other value is rejected.
+Other algorithms (lz4, zstd) are planned but not implemented. Any other value is rejected.
## Message batch format
@@ -505,7 +507,7 @@ frame = [frame header: 48 bytes][payload][user_headers]
| 36 | 4 | `payload_length` | u32 | Byte length of the payload. |
| 40 | 8 | reserved | u64 | Must be zero. |
-After the 48 header bytes: `payload` (`payload_length` bytes), then `user_headers` (`user_headers_length` bytes). Note the payload comes **first**.
+After the 48 header bytes: `payload` (`payload_length` bytes), then `user_headers` (`user_headers_length` bytes). The payload comes **first**.
A frame's absolute position is derived, not stored: offset = `base_offset + offset_delta`, timestamp = `base_timestamp + timestamp_delta` for the server-stamped time (`origin_timestamp + timestamp_delta` for producer time).
@@ -607,7 +609,7 @@ Patch semantics: absent option keys are left unchanged.
[stream_id: Identifier][partitions_count: u32][name_len: u8][name: N][options block to end]
```
-The fixed fields are the shape of the operation: which stream, how many partitions, what name. Every topic setting (`compression_algorithm`, `message_expiry`, `max_topic_size`, `segment_size`, `enforce_fsync`, `messages_required_to_save`, `size_of_messages_required_to_save`, `preallocate_segments`) rides the options block; see [Topic options](/docs/server/topic-options). `partitions_count` is an argument, not a setting: it is consumed at admission and never persisted as an option.
+The fixed fields are the shape of the operation: which stream, how many partitions, what name. Every topic setting (`compression_algorithm`, `message_expiry`, `max_topic_size`, `segment_size`, `enforce_fsync`, `messages_required_to_save`, `size_of_messages_required_to_save`, `preallocate_segments`) rides the options block. See [Topic options](/docs/server/topic-options). `partitions_count` is an argument, not a setting: it is consumed at admission and never persisted as an option.
**Delete topic. Code: 303.**
@@ -621,7 +623,7 @@ The fixed fields are the shape of the operation: which stream, how many partitio
[stream_id: Identifier][topic_id: Identifier][name_len: u8][name: N][options block to end]
```
-Only `compression_algorithm`, `message_expiry`, and `max_topic_size` are updatable; absent keys are left unchanged.
+Only `compression_algorithm`, `message_expiry`, and `max_topic_size` are updatable. Absent keys are left unchanged.
**Purge topic. Code: 305.**
@@ -683,7 +685,7 @@ The 16-byte prefix is followed by a stream of [batch records](#message-batch-for
[batch: 256-byte batch header + frames]
```
-`metadata_length` counts the bytes from `stream_id` through `messages_count` inclusive, so a reader can skip straight to the batch. The producer leaves `partition_id`, `base_offset`, and `base_timestamp` zero in the batch header; the server stamps them. Every checksum is producer-computed and verified at admission; the reserved regions must be zero.
+`metadata_length` counts the bytes from `stream_id` through `messages_count` inclusive, so a reader can skip straight to the batch. The producer leaves `partition_id`, `base_offset`, and `base_timestamp` zero in the batch header, and the server stamps them. Every checksum is producer-computed and verified at admission. The reserved regions must be zero.
**Flush unsaved buffer. Code: 102.**
@@ -722,7 +724,7 @@ Response body (20 bytes):
[partition_flag: u8][partition_id: u32][ack: u8]
```
-The trailing `ack` byte is mandatory on both write commands; a payload without it fails to decode. Ack levels:
+The trailing `ack` byte is mandatory on both write commands. A payload without it fails to decode. Ack levels:
| Value | Level | Meaning |
|-------|-------|---------|
@@ -793,7 +795,7 @@ Read-only: a member asks for its current partition assignment and the group gene
[options block to end]
```
-`permissions_len` and the permissions bytes are present **only** when `has_permissions = 1`; when it is 0 the options block follows immediately. The user option catalog is empty today, so the block is normally empty. `status`: 1 = active, 2 = inactive.
+`permissions_len` and the permissions bytes are present **only** when `has_permissions = 1`. When it is 0 the options block follows immediately. The user option catalog is empty today, so the block is normally empty. `status`: 1 = active, 2 = inactive.
**Delete user. Code: 34.**
@@ -831,10 +833,10 @@ As in `CreateUser`, the length field is conditional: with `has_permissions = 0`
Login and logout are part of the [connection lifecycle](#connection-lifecycle):
-- **Login register. Code: 40.** `Operation::Register`; body is `ClientVersionInfo` + credentials, described [above](#login-register).
-- **Login register with PAT. Code: 45.** `Operation::Register`; body is `ClientVersionInfo` + token.
-- **Logout user. Code: 39.** `Operation::Logout`; empty payload.
-- Codes **38** and **44** are the pre-register login shapes; the server rejects both with a `MalformedLogin` eviction.
+- **Login register. Code: 40.** `Operation::Register`. The body is `ClientVersionInfo` + credentials, described [above](#login-register).
+- **Login register with PAT. Code: 45.** `Operation::Register`. The body is `ClientVersionInfo` + token.
+- **Logout user. Code: 39.** `Operation::Logout`. Empty payload.
+- Codes **38** and **44** are the pre-register login shapes. The server rejects both with a `MalformedLogin` eviction.
### Personal access tokens
diff --git a/content/docs/server/security.mdx b/content/docs/server/security.mdx
index e8def0ce..1294b2c3 100644
--- a/content/docs/server/security.mdx
+++ b/content/docs/server/security.mdx
@@ -21,7 +21,7 @@ Or use the `--with-default-root-credentials` flag for development (sets root cre
**Important**: once the data directory exists, environment variable credentials are **ignored**. To reset credentials, you must use the `--fresh` flag (which deletes all data).
-The root user **cannot be deleted**, and its permissions cannot be changed: it always holds every permission.
+The root user **cannot be deleted**, and its permissions are fixed: it always holds every permission.
### Personal Access Tokens (PAT)
@@ -89,7 +89,7 @@ Permissions are checked from top to bottom: global, then stream, then topic. A p
If a stream has no entry in the user's stream permissions, only global permissions apply to it. The same holds for topics within a stream.
-For example, a user that may only consume from stream 42 needs no global permissions at all: grant stream-scoped `read_stream` and `poll_messages` on stream 42, and the user can read that stream, list its topics, and poll messages from any topic in it, nothing else.
+For example, a user that may only consume from stream 42 needs no global permissions at all: grant stream-scoped `read_stream` and `poll_messages` on stream 42, and the user can read that stream, list its topics, and poll messages from any topic in it. Nothing else.
## Transport encryption (TLS)
@@ -102,7 +102,7 @@ Each transport configures TLS in its own section, and the `self_signed` semantic
| QUIC | `[quic.certificate]` | TLS is mandatory (part of the QUIC spec). `self_signed = true` **always** generates an ephemeral certificate and ignores `cert_file`/`key_file`, logging a warning if the files exist. |
| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires real `cert_file`/`key_file`. |
-For production deployments, provide proper certificates via `cert_file` and `key_file`; ephemeral certificates change on every start and cannot be verified by clients. See [Networking](/docs/server/networking) for the surrounding transport configuration.
+For production deployments, provide proper certificates via `cert_file` and `key_file`. Ephemeral certificates change on every start and cannot be verified by clients. See [Networking](/docs/server/networking) for the surrounding transport configuration.
## Data encryption at rest
@@ -129,11 +129,11 @@ clock_skew = "5 s"
Further keys (`valid_issuers`, `valid_audiences`, `not_before`, `use_base64_secret`, the signing secrets) are covered in [Configuration](/docs/server/configuration).
-**Signing secrets**: `encoding_secret` and `decoding_secret` default to empty, which makes the server generate a secure random secret on every start. That is a safe single-node default with two consequences: issued tokens **die on restart**, and in a cluster each node signs with its own key, so bearers are node-local and follower-to-primary request forwarding stays disabled. For clusters, configure an identical secret on every node (prefer the `IGGY_HTTP_JWT_ENCODING_SECRET`/`IGGY_HTTP_J [...]
+**Signing secrets**: `encoding_secret` and `decoding_secret` default to empty, which makes the server generate a secure random secret on every start. That's a safe single-node default with two consequences: issued tokens **die on restart**, and in a cluster each node signs with its own key, so bearers are node-local and follower-to-primary request forwarding stays disabled. For clusters, configure an identical secret on every node (prefer the `IGGY_HTTP_JWT_ENCODING_SECRET`/`IGGY_HTTP_JW [...]
**Refresh tokens**: `POST /users/refresh-token` re-issues an access token from a still-valid one presented in the request body, answering the same identity shape as login, so HTTP clients can extend a session without re-sending credentials.
-**Federated issuers**: the opt-in `[[http.jwt.trusted_issuers]]` list accepts tokens minted by external identity providers for application-to-application flows. Each entry names an `issuer`, `audience`, and `jwks_url`; signatures are verified against the issuer's JWKS (fetches are rate-limited), and every token from that issuer is remapped onto the configured non-root `user_id`. With no entries configured, the listener accepts only self-issued tokens. See [Configuration](/docs/server/con [...]
+**Federated issuers**: the opt-in `[[http.jwt.trusted_issuers]]` list accepts tokens minted by external identity providers for application-to-application flows. Each entry names an `issuer`, `audience`, and `jwks_url`. Signatures are verified against the issuer's JWKS (fetches are rate-limited), and every token from that issuer is remapped onto the configured non-root `user_id`. With no entries configured, the listener accepts only self-issued tokens. See [Configuration](/docs/server/con [...]
## Cluster security
diff --git a/content/docs/server/storage-engine.mdx b/content/docs/server/storage-engine.mdx
index 503087a5..c56fb160 100644
--- a/content/docs/server/storage-engine.mdx
+++ b/content/docs/server/storage-engine.mdx
@@ -40,7 +40,7 @@ local_data/
└── 00000000000016000000.index
```
-Stream, topic, and partition ids are numeric and **0-based**. Each partition directory holds pairs of `.log` and `.index` files; the filename is the start offset of the segment's first message, zero-padded to 20 digits. Next to the segments, every partition keeps its own superblock pair (replica identity and consensus state for that partition) and an `offsets/` tree for consumer offset storage.
+Stream, topic, and partition ids are numeric and **0-based**. Each partition directory holds pairs of `.log` and `.index` files. The filename is the start offset of the segment's first message, zero-padded to 20 digits. Next to the segments, every partition keeps its own superblock pair (replica identity and consensus state for that partition) and an `offsets/` tree for consumer offset storage.
## Segmented log
@@ -48,7 +48,7 @@ Stream, topic, and partition ids are numeric and **0-based**. Each partition dir
Each partition is a segmented log: an ordered list of **sealed** (read-only) segments plus one **active** (writable) segment. When the active segment reaches the topic's `segment_size` (default 1 GiB), it is sealed and a new active segment starts at the next offset.
-`segment_size` is a per-topic creation option, bounded to a 512-byte multiple between 1 MiB and 1 GiB. It is a **soft limit**: rotation fires after the append that crosses it, so the crossing batch lands whole and a sealed segment may run one batch past the configured size. `preallocate_segments` (also per topic) reserves each segment's bytes up front on filesystems that support it. See [Topic options](/docs/server/topic-options).
+`segment_size` is a per-topic creation option, bounded to a 512-byte multiple between 1 MiB and 1 GiB. It's a **soft limit**: rotation fires after the append that crosses it, so the crossing batch lands whole and a sealed segment may run one batch past the configured size. `preallocate_segments` (also per topic) reserves each segment's bytes up front on filesystems that support it. See [Topic options](/docs/server/topic-options).
## On-disk batch format
@@ -59,9 +59,9 @@ Segments store **batch records**, not individual messages. The record is byte-id
frame = [frame header: 48 bytes][payload][user_headers]
```
-The 256-byte batch header carries `partition_id`, `base_offset`, `base_timestamp`, `origin_timestamp`, `batch_length`, `batch_checksum` (XxHash3-64), and `message_count`; the rest is reserved and must be zero. Each 48-byte frame header carries `checksum` (XxHash3-64), `id` (u128), `offset_delta` (u32), `timestamp_delta` (u32), and the two lengths. A message's absolute offset is `base_offset + offset_delta`; timestamps resolve the same way against the base. All fields are **little-endian* [...]
+The 256-byte batch header carries `partition_id`, `base_offset`, `base_timestamp`, `origin_timestamp`, `batch_length`, `batch_checksum` (XxHash3-64), and `message_count`. The rest is reserved and must be zero. Each 48-byte frame header carries `checksum` (XxHash3-64), `id` (u128), `offset_delta` (u32), `timestamp_delta` (u32), and the two lengths. A message's absolute offset is `base_offset + offset_delta`, and timestamps resolve the same way against the base. All fields are **little-end [...]
-The full byte-level tables live on the [Schema](/docs/server/schema#message-batch-format) page; what is written to disk is exactly what the producer sent, with the server stamping `partition_id` at ingestion and `base_offset` / `base_timestamp` (plus a `batch_checksum` recompute) at persistence.
+The full byte-level tables live on the [Schema](/docs/server/schema#message-batch-format) page. What lands on disk is exactly what the producer sent, with the server stamping `partition_id` at ingestion and `base_offset` / `base_timestamp` (plus a `batch_checksum` recompute) at persistence.
## Indexes
@@ -108,7 +108,7 @@ The flush thresholds and fsync policy are **per-topic creation options** (they u
Whichever threshold trips first flushes. These are soft limits: a flush writes whole batches, so the actual flushed count or size can overshoot. The `MessagesWriter` uses **vectored I/O** (`writev`) with up to `MAX_IOV_COUNT = 1024` buffers per syscall, so many buffered batches land in one write.
-On replication: every partition belongs to a consensus group (Viewstamped Replication) and writes are prepared and quorum-acknowledged before commit. At HEAD the partition-plane consensus journal is **in-memory**; durability of message data comes from segment persistence and, in clusters, from the copies on other replicas. The metadata plane (below) has a durable on-disk journal.
+Every partition belongs to a consensus group (Viewstamped Replication), and writes are prepared and quorum-acknowledged before commit. At HEAD the partition-plane consensus journal is **in-memory**. Durability of message data comes from segment persistence and, in clusters, from the copies on other replicas. The metadata plane (below) has a durable on-disk journal.
## Boot-time segment recovery
@@ -116,7 +116,7 @@ At startup the server recovers every partition from its files, tolerating a torn
1. Sweep leftovers: `.staging` files and orphan `.index` files (an index without its `.log`) are deleted. A `.log` is **never deleted**.
2. For each segment, read the segment's bounds (offsets, timestamps, size) from the 24-byte sparse index.
-3. If the index is missing or torn, walk the `.log` batch by batch and recover the bounds from the records themselves. This is not tail-only: with the default `enforce_fsync = false`, a torn index can appear mid-chain, and the log walk recovers it.
+3. If the index is missing or torn, walk the `.log` batch by batch and recover the bounds from the records themselves. This isn't tail-only: with the default `enforce_fsync = false`, a torn index can appear mid-chain, and the log walk recovers it.
4. If neither the index nor the log holds one whole batch, the segment is recovered as empty: sizes are zeroed so the next append overwrites the torn bytes instead of stranding undecodable garbage inside the readable range.
5. The last segment is left unsealed and becomes the active segment.
@@ -131,13 +131,13 @@ Disk reads are verified before they reach a consumer:
validate_checksum = true
```
-With `validate_checksum = true` (the default), every batch a disk poll reads is re-hashed and compared against its stored checksum; a mismatch **fails the poll closed**, so a segment damaged at rest is reported instead of served. Setting it to `false` skips the re-hash and serves whatever decodes, which can hand a consumer bytes provably not the ones written; only disable it with a corruption guard elsewhere in the stack.
+With `validate_checksum = true` (the default), every batch a disk poll reads is re-hashed and compared against its stored checksum. A mismatch **fails the poll closed**, so a segment damaged at rest is reported instead of served. Setting it to `false` skips the re-hash and serves whatever decodes, which can hand a consumer bytes provably not the ones written. Only disable it with a corruption guard elsewhere in the stack.
Polls serve stored batch records as-is (a reply may be a server-sliced view of a larger stored batch), so there is no re-encoding on the read path.
## Memory pool
-Iggy includes a custom memory pool to eliminate allocation overhead on the hot path. The pool has **28 buckets** with buffer sizes from 4 KiB up to 512 MiB (non-uniform spacing, denser around common message sizes; above 2 MiB sizes round to hugepage-friendly steps). Components request a buffer from the appropriate bucket and return it when done.
+Iggy includes a custom memory pool to eliminate allocation overhead on the hot path. The pool has **28 buckets** with buffer sizes from 4 KiB up to 512 MiB (non-uniform spacing, denser around common message sizes, with sizes above 2 MiB rounded to hugepage-friendly steps). Components request a buffer from the appropriate bucket and return it when done.
```toml
[system.memory_pool]
@@ -150,7 +150,7 @@ This avoids heap allocations during message processing and enables zero-copy mes
## Retention and cleanup
-Two independent retention policies exist, both **per-topic creation options** (see [Topic options](/docs/server/topic-options)); the segment cleaner enforces them:
+Two independent retention policies exist, both **per-topic creation options** (see [Topic options](/docs/server/topic-options)). The segment cleaner enforces them:
```toml
[data_maintenance.messages]
@@ -158,11 +158,11 @@ cleaner_enabled = true # default
interval = "1 m" # default
```
-**Size-based retention** (`max_topic_size`): the cluster has no single owner of a topic-wide total, so each partition enforces an equal share: `max_topic_size / partition_count`, counted over **sealed** bytes. The share is floored at one sealed-segment ceiling (`segment_size` plus the maximum message-bus frame, since a sealed segment can overshoot by one batch); a cap that divides below that floor is raised to it, so the policy always means at least "keep the newest sealed segment". Once [...]
+**Size-based retention** (`max_topic_size`): the cluster has no single owner of a topic-wide total, so each partition enforces an equal share: `max_topic_size / partition_count`, counted over **sealed** bytes. The share is floored at one sealed-segment ceiling (`segment_size` plus the maximum message-bus frame, since a sealed segment can overshoot by one batch). A cap that divides below that floor is raised to it, so the policy always means at least "keep the newest sealed segment". Once [...]
**Time-based retention** (`message_expiry`): sealed segments whose newest message is older than the expiry are deleted.
-Both policies can be active at once. They only ever touch sealed segments; the active segment is **never deleted**, even if its messages have expired.
+Both policies can be active at once. They only ever touch sealed segments. The active segment is **never deleted**, even if its messages have expired.
## Metadata plane
@@ -170,7 +170,7 @@ Metadata operations (create stream, delete topic, create user, and so on) do not
- **`journal.wal`**: the prepare journal. Each entry is a 256-byte `PrepareHeader` (the consensus header carrying checksums, view, op and commit numbers, operation discriminant, and acting user) followed by the operation's wire-format body, exactly as replicated. This is the write-ahead log: an operation is journaled before it is applied.
- **`snapshot.bin`**: a **MessagePack** (rmp-serde) snapshot of the whole metadata state machine, written at checkpoints so the journal can be truncated. Recovery loads the snapshot, then replays journal entries newer than it, rebuilding the client session table alongside.
-- **`superblock.a` / `superblock.b`**: a ping-pong pair of checkpoint records; the newest valid slot wins by sequence number. The superblock records which checkpoint is current and the snapshot's checksum. Boot cross-checks the pairing: a superblock pointing at a checkpoint newer than the on-disk snapshot, or a snapshot whose checksum does not match the record, refuses to boot rather than silently rewinding committed state.
+- **`superblock.a` / `superblock.b`**: a ping-pong pair of checkpoint records, where the newest valid slot wins by sequence number. The superblock records which checkpoint is current and the snapshot's checksum. Boot cross-checks the pairing: a superblock pointing at a checkpoint newer than the on-disk snapshot, or a snapshot whose checksum doesn't match the record, refuses to boot rather than silently rewinding committed state.
Partition directories carry the same superblock mechanism for their own consensus state (replica identity, view), which is how a restarted replica proves it is itself rather than a blank impostor.
@@ -194,7 +194,7 @@ Earlier releases documented several storage features and `[system.*]` keys that
- **`cache_indexes`**: removed. Index caching is internal now (see [Indexes](#indexes)).
- **`[system.message_deduplication]`**: the server-side deduplicator was removed entirely.
-- **Segment archiving / S3 backup**: removed. `[system.segment] archive_expired` must stay `false`; setting it to `true` aborts boot.
+- **Segment archiving / S3 backup**: removed. `[system.segment] archive_expired` must stay `false`. Setting it to `true` aborts boot.
- **`[system.topic]` / `[system.partition]` / `[system.segment]` storage knobs** (`max_size`, `message_expiry`, `enforce_fsync`, `messages_required_to_save`, `size_of_messages_required_to_save`, `size`, `preallocate`): relocated to per-topic options.
The config loader **refuses to boot** while any relocated or removed key is still set, in the file or the environment, so stale configs fail fast instead of silently ignoring a knob. The full mapping is in [Relocated configuration keys](/docs/server/configuration#relocated-configuration-keys).
diff --git a/content/docs/server/topic-options.mdx b/content/docs/server/topic-options.mdx
index 2c97efac..9214b347 100644
--- a/content/docs/server/topic-options.mdx
+++ b/content/docs/server/topic-options.mdx
@@ -2,9 +2,9 @@
title: Topic Options
---
-Retention, durability, and segment layout are per-topic decisions, made when the topic is created. They used to be server-wide `[system.*]` keys, which gave every knob two homes; the server now [refuses to boot](/docs/server/configuration#relocated-configuration-keys) on the old keys and each topic carries its own values instead.
+Retention, durability, and segment layout are per-topic decisions, made when the topic is created. They used to be server-wide `[system.*]` keys, which gave every knob two homes. The server now [refuses to boot](/docs/server/configuration#relocated-configuration-keys) on the old keys, and each topic carries its own values instead.
-Options are key-value pairs sent with `CreateTopic`. Unknown keys are **rejected at the edge, never skipped**: a silently ignored knob would hand the client server defaults without it ever learning. The same mechanism exists for streams and users, but their catalogs are empty today; only topics have option keys.
+Options are key-value pairs sent with `CreateTopic`. Unknown keys are **rejected at the edge, never skipped**: a silently ignored knob would leave the client on server defaults with no way to find out. The same mechanism exists for streams and users, but their catalogs are empty today. Only topics have option keys.
## The catalog
@@ -19,11 +19,11 @@ Options are key-value pairs sent with `CreateTopic`. Unknown keys are **rejected
| `size_of_messages_required_to_save` | 1 MiB | at most 1 GiB | Flush the journal once it holds this many bytes. Paired with the message count; whichever threshold trips first flushes. |
| `preallocate_segments` | `false` | `segment_size` x partitions at most 64 GiB per create | Reserve each segment's bytes up front where the filesystem supports it. |
-Both retention policies can be active at once. The active segment is **never touched**; deletion is done by the server's segment cleaner (`[data_maintenance.messages]`, enabled by default).
+Both retention policies can be active at once. The active segment is **never touched**. Deletion is done by the server's segment cleaner (`[data_maintenance.messages]`, enabled by default).
Value forms are forgiving: byte sizes accept a raw number of bytes or a string like `"128 MiB"`, expiry accepts microseconds or a humantime string like `"7 days"`, booleans accept `true`/`false`. Create admission re-parses and re-encodes what you send, so a string `segment_size=128MiB` is stored as the number it names.
-A note on `preallocate_segments`: it reserves exactly `segment_size` of real disk per partition the moment the topic is created (and again as segments rotate). With the default 1 GiB segment size that is 1 GiB per partition up front, which is why it is opt-in and why one create is **capped at 64 GiB** of total reservation.
+`preallocate_segments` reserves exactly `segment_size` of real disk per partition the moment the topic is created (and again as segments rotate). With the default 1 GiB segment size that's 1 GiB per partition up front, which is why it's opt-in and why one create is **capped at 64 GiB** of total reservation.
## Setting options
@@ -37,11 +37,11 @@ iggy topic create my-stream my-topic 1 none \
--set enforce_fsync=true
```
-In the CLI, `compression_algorithm`, `message_expiry`, and `max_topic_size` also have first-class parameters on `iggy topic create`; `--set` covers the storage knobs that have no named parameter of their own. Values are sent as strings and parsed server-side, so `iggy options topic` (below) tells you exactly what this server accepts.
+In the CLI, `compression_algorithm`, `message_expiry`, and `max_topic_size` also have first-class parameters on `iggy topic create`. `--set` covers the storage knobs that have no named parameter of their own. Values are sent as strings and parsed server-side, so `iggy options topic` (below) tells you exactly what this server accepts.
SDKs pass options as a parameter on the create call (`TopicCreateOptions::raw` in Rust, an `options` dictionary or list in Node, Python, Go, Java, C#, and C++). The HTTP API takes them as a plain string map in the create body.
-Keys you do not send are resolved to the defaults above by the admitting server and stored as **derived** entries. `GetTopic` returns both blocks, explicit and derived, so the effective value of every knob is always visible, along with who chose it.
+Keys you don't send are resolved to the defaults above by the admitting server and stored as **derived** entries. `GetTopic` returns both blocks, explicit and derived, so the effective value of every knob is always visible, along with who chose it.
## Create-only vs updatable
@@ -51,7 +51,7 @@ Keys you do not send are resolved to the defaults above by the admitting server
- `message_expiry`
- `max_topic_size`
-Updates are **patches**: a key you do not send keeps its current value.
+Updates are **patches**: a key you don't send keeps its current value.
The storage knobs (`segment_size`, `enforce_fsync`, `messages_required_to_save`, `size_of_messages_required_to_save`, `preallocate_segments`) are **create-only**. They describe how a partition's storage is laid down: changing `segment_size` mid-segment would leave segments sized by different caps, and preallocation can only act on a file not yet opened. A topic gets them at creation and keeps them, so its segments stay uniform. This is also why the server refuses to boot on the old confi [...]
@@ -73,6 +73,6 @@ Discovery matters most on the binary transports: TCP, QUIC, and WebSocket carry
## Durability
-Single-node durability is a per-topic decision. For the strongest guarantee, create the topic with `enforce_fsync=true`; every write is then **fsynced before it is acknowledged**. Without it, the flush thresholds (`messages_required_to_save`, `size_of_messages_required_to_save`) bound how much buffered data a crash can lose, at much higher throughput.
+Single-node durability is a per-topic decision. For the strongest guarantee, create the topic with `enforce_fsync=true`. Every write is then **fsynced before it is acknowledged**. Without it, the flush thresholds (`messages_required_to_save`, `size_of_messages_required_to_save`) bound how much buffered data a crash can lose, at much higher throughput.
In cluster mode, durability comes from quorum replication: a write is acknowledged once a majority of replicas hold it. See [Clustering](/docs/clustering/vsr).
diff --git a/content/docs/web_ui/start.mdx b/content/docs/web_ui/start.mdx
index 3caa8201..c43cf713 100644
--- a/content/docs/web_ui/start.mdx
+++ b/content/docs/web_ui/start.mdx
@@ -11,7 +11,7 @@ Iggy Web UI provides a comprehensive dashboard for Iggy server. It allows you to
The Web UI is a purely client-side application: every call to the Iggy HTTP API is issued by your browser, not by the Web UI container or server. Two things follow from this:
- `PUBLIC_IGGY_API_URL` must be a URL that the **browser** can reach (e.g. `http://localhost:3010`), not an internal Docker network hostname like `http://iggy:80`.
-- The Iggy server must allow cross-origin requests from the Web UI origin. CORS is enabled for all origins by default (`[http.cors]` in the server config); if you restricted `allowed_origins`, add the Web UI origin.
+- The Iggy server must allow cross-origin requests from the Web UI origin. CORS is enabled for all origins by default (`[http.cors]` in the server config). If you restricted `allowed_origins`, add the Web UI origin.
In embedded mode neither applies: the UI is served from the same origin as the HTTP API and uses relative URLs.
@@ -38,11 +38,11 @@ docker pull apache/iggy-web-ui
docker run -e PUBLIC_IGGY_API_URL=http://localhost:3000 -p 3050:3050 apache/iggy-web-ui
```
-Remember that `PUBLIC_IGGY_API_URL` is consumed by the browser: if the Iggy server also runs in Docker, use the host-published HTTP port here, not the container's network alias.
+`PUBLIC_IGGY_API_URL` is consumed by the browser: if the Iggy server also runs in Docker, use the host-published HTTP port here, not the container's network alias.
### Local development
-The Web UI lives in the `web/` folder of the [repository](https://github.com/apache/iggy/tree/master/web). Use a Node.js version supported by the frontend toolchain (`^20.19.0 || >=22.12.0` per Vite; 22 LTS is a safe default) and `npm` (`pnpm` and `yarn` are not part of the supported workflow):
+The Web UI lives in the `web/` folder of the [repository](https://github.com/apache/iggy/tree/master/web). Use a Node.js version supported by the frontend toolchain (Vite requires `^20.19.0 || >=22.12.0`, so 22 LTS is a safe default) and `npm` (`pnpm` and `yarn` are not part of the supported workflow):
```bash
cd web
@@ -56,7 +56,7 @@ Point the UI at your Iggy server's HTTP API via a `.env` file in the `web/` fold
The Web UI provides the following pages and functionality:
-- **Overview** - server health overview and key metrics
+- **Overview** - server health and key metrics
- **Streams** - list, create, and manage streams
- **Topics** - manage topics per stream (create, update, delete, purge)
- **Partitions** - view partition details per topic
@@ -67,7 +67,7 @@ The Web UI provides the following pages and functionality:
- **Users** - user management (create, update, delete, permissions)
- **Settings** - Web UI preferences
-Some pages visible in the navigation (Clients, Logs, terminal, server settings) are placeholders that are not functional yet; see the [roadmap](https://github.com/apache/iggy/tree/master/web#roadmap) for what is planned.
+Some pages visible in the navigation (Clients, Logs, terminal, server settings) are placeholders that aren't functional yet. See the [roadmap](https://github.com/apache/iggy/tree/master/web#roadmap) for what is planned.
## Docker Compose example
diff --git a/src/app/(home)/page.tsx b/src/app/(home)/page.tsx
index 78e641cd..7fd815b0 100644
--- a/src/app/(home)/page.tsx
+++ b/src/app/(home)/page.tsx
@@ -41,7 +41,7 @@ const heroStats = [
gradient: "from-[#5f87fd] to-[#7d44e0]",
},
{
- value: "7",
+ value: "8",
label: "Language SDKs",
gradient: "from-[#9d44e0] to-[#e55efa]",
},
@@ -71,7 +71,7 @@ const features = [
{
title: "Multi-Language SDKs",
description:
- "Client libraries available for Rust, C#, Java, Go, Python, Node.js and PHP, with C++ coming soon.",
+ "Client libraries available for Rust, C#, C++, Java, Go, Python, Node.js and PHP.",
},
{
title: "Consumer Groups & Partitioning",
@@ -230,21 +230,22 @@ export default function HomePage() {
<div className="mx-auto mb-14 max-w-5xl border-y border-white/[0.1]">
<div className="grid gap-5 py-6 md:grid-cols-[auto_1fr_auto] md:items-center md:gap-7">
<span className="w-fit rounded-md border border-[#38bdf8]/30 bg-[#38bdf8]/10 px-2.5 py-1 font-mono text-xs font-semibold uppercase text-[#7dd3fc]">
- Experimental
+ Shipped
</span>
<div className="min-w-0">
<h3 className="text-lg font-semibold text-[#fffaeb]">
- VSR clustering is coming soon
+ VSR clustering is built in
</h3>
<p className="mt-1 text-sm leading-relaxed text-[#aeb5bd]">
- Viewstamped Replication Revisited is already implemented in
- the{" "}
- <span className="font-mono text-[#d7dce1]">server-ng</span>{" "}
- module on main. Its deterministic simulation testing (DST)
- exercises failures, delays, restarts and network partitions to
- validate consensus. Together, VSR and DST provide the
- foundation for highly available, fault-tolerant and reliable
- Iggy clusters.
+ Viewstamped Replication Revisited is built into the server:
+ it runs as a single node by default, and multi-node
+ clustering is enabled through the{" "}
+ <span className="font-mono text-[#d7dce1]">[cluster]</span>{" "}
+ configuration section. Its deterministic simulation testing
+ (DST) exercises failures, delays, restarts and network
+ partitions to validate consensus. Together, VSR and DST
+ provide the foundation for highly available, fault-tolerant
+ and reliable Iggy clusters.
</p>
</div>
<Link
diff --git a/src/app/docs/[[...slug]]/page.tsx b/src/app/docs/[[...slug]]/page.tsx
index 0c784c23..75f345f9 100644
--- a/src/app/docs/[[...slug]]/page.tsx
+++ b/src/app/docs/[[...slug]]/page.tsx
@@ -28,6 +28,7 @@ import { notFound } from "next/navigation";
import { getMDXComponents } from "@/mdx-components";
import type { Metadata } from "next";
import { createRelativeLink } from "fumadocs-ui/mdx";
+import * as PageTree from "fumadocs-core/page-tree";
export default async function Page(props: {
params: Promise<{ slug?: string[] }>;
@@ -37,9 +38,20 @@ export default async function Page(props: {
if (!page) notFound();
const MDX = page.data.body;
+ const tree = source.getPageTree();
+ const neighbours = PageTree.findNeighbour(tree, page.url);
return (
- <DocsPage toc={page.data.toc} full={page.data.full}>
+ <DocsPage
+ toc={page.data.toc}
+ full={page.data.full}
+ footer={{
+ items: {
+ previous: withChapter(tree, neighbours.previous),
+ next: withChapter(tree, neighbours.next),
+ },
+ }}
+ >
<DocsTitle>{page.data.title}</DocsTitle>
<DocsDescription>{page.data.description}</DocsDescription>
<DocsBody>
@@ -69,3 +81,24 @@ export async function generateMetadata(props: {
description: page.data.description,
};
}
+
+// Several chapters have pages with identical titles ("Introduction",
+// "Examples"), so a bare title in the prev/next footer is ambiguous.
+// Prefix it with the chapter, i.e. the target's nearest parent folder.
+function withChapter(tree: PageTree.Root, item: PageTree.Item | undefined) {
+ if (!item) return undefined;
+ const parent = PageTree.findParent(tree, item.url);
+ // The tree root is not a chapter; top-level pages keep a bare title.
+ if (!parent || !("type" in parent)) return item;
+ return {
+ ...item,
+ name: (
+ <>
+ <span className="me-1 font-normal text-fd-muted-foreground">
+ {parent.name}:
+ </span>
+ {item.name}
+ </>
+ ),
+ };
+}
diff --git a/src/app/global.css b/src/app/global.css
index edfbf3b5..ff0811a1 100644
--- a/src/app/global.css
+++ b/src/app/global.css
@@ -77,6 +77,12 @@ header#nd-nav button {
font-size: 0.9375rem;
}
+/* The default fumadocs mobile navbar background is only 80% opaque, which
+ lets page content ghost through while scrolling underneath. Force it opaque. */
+header#nd-subnav {
+ background-color: var(--color-fd-background) !important;
+}
+
figure.shiki {
border-color: var(--color-fd-border) !important;
box-shadow: none !important;
diff --git a/src/components/architecture-diagrams.tsx b/src/components/architecture-diagrams.tsx
index e993526f..c4d025aa 100644
--- a/src/components/architecture-diagrams.tsx
+++ b/src/components/architecture-diagrams.tsx
@@ -1047,14 +1047,14 @@ export function WhyIggy() {
<p className="text-xs text-fd-muted-foreground m-0 mb-5">How Iggy compares to traditional message streaming platforms</p>
<div className="space-y-2">
- <div className="grid grid-cols-[100px_1fr_1fr] md:grid-cols-[120px_1fr_1fr] gap-2 mb-1">
- <span className="text-[10px] font-semibold text-fd-muted-foreground" />
+ <div className="grid grid-cols-2 sm:grid-cols-[100px_1fr_1fr] md:grid-cols-[120px_1fr_1fr] gap-2 mb-1">
+ <span className="hidden sm:block text-[10px] font-semibold text-fd-muted-foreground" />
<span className="text-[10px] font-semibold text-fd-muted-foreground text-center">Traditional</span>
<span className="text-[10px] font-semibold text-fd-primary text-center">Iggy</span>
</div>
{comparisons.map((c) => (
- <div key={c.label} className="grid grid-cols-[100px_1fr_1fr] md:grid-cols-[120px_1fr_1fr] gap-2 items-center">
- <span className="text-[11px] font-semibold text-fd-foreground">{c.label}</span>
+ <div key={c.label} className="grid grid-cols-2 sm:grid-cols-[100px_1fr_1fr] md:grid-cols-[120px_1fr_1fr] gap-2 items-center">
+ <span className="col-span-2 sm:col-span-1 text-[11px] font-semibold text-fd-foreground">{c.label}</span>
<div className="rounded-md bg-fd-muted/30 px-3 py-2 text-center">
<span className="text-[10px] text-fd-muted-foreground">{c.traditional}</span>
</div>
@@ -1243,7 +1243,7 @@ export function DocsHero() {
</div>
</div>
- <div className="flex justify-center gap-5 mt-4 pt-3 border-t border-fd-border/50">
+ <div className="grid grid-cols-2 gap-3 sm:flex sm:justify-center sm:gap-5 mt-4 pt-3 border-t border-fd-border/50">
{[
{ label: "MCP Server", href: "/docs/ai/mcp", sub: "40+ LLM tools" },
{ label: "Web UI", href: "/docs/web_ui/start", sub: "Dashboard" },
diff --git a/src/components/code-tabs.tsx b/src/components/code-tabs.tsx
index ccfd9315..c3a0be28 100644
--- a/src/components/code-tabs.tsx
+++ b/src/components/code-tabs.tsx
@@ -30,7 +30,7 @@ const pkgInfo: Record<string, { install: string; url: string; label: string }> =
"Node.js": { install: "npm install apache-iggy", url: "https://www.npmjs.com/package/apache-iggy", label: "npm" },
"C#": { install: "dotnet add package Apache.Iggy", url: "https://www.nuget.org/packages/Apache.Iggy/", label: "NuGet" },
"PHP": { install: "cargo php install --release --yes", url: "https://github.com/apache/iggy/tree/master/foreign/php", label: "GitHub" },
- "C++ (WIP)": { install: "git clone https://github.com/apache/iggy", url: "https://github.com/apache/iggy/tree/master/foreign/cpp", label: "GitHub" },
+ "C++": { install: "git clone https://github.com/apache/iggy", url: "https://github.com/apache/iggy/tree/master/foreign/cpp", label: "GitHub" },
};
const snippets = [
@@ -244,7 +244,7 @@ $client->sendMessages(
);`,
},
{
- lang: "C++ (WIP)",
+ lang: "C++",
file: "producer.cpp",
href: "/docs/sdk/cpp/intro",
code: `#include "lib.rs.h"