You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by pi...@apache.org on 2025/04/21 11:18:22 UTC
(iggy) branch master updated: chore(docs): update readme (#1716)
This is an automated email from the ASF dual-hosted git repository.
piotr pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iggy.git
The following commit(s) were added to refs/heads/master by this push:
new 25782a32 chore(docs): update readme (#1716)
25782a32 is described below
commit 25782a32dd578f210362105c4c989513ddf357d1
Author: Piotr Gankiewicz <pi...@gmail.com>
AuthorDate: Mon Apr 21 13:18:18 2025 +0200
chore(docs): update readme (#1716)
---
README.md | 89 ++++++++++++++++++++++++++++---------------
assets/bench.png | Bin 0 -> 380000 bytes
assets/cli.png | Bin 354629 -> 231092 bytes
assets/files_structure.png | Bin 223296 -> 201217 bytes
assets/iggy_architecture.png | Bin 0 -> 760914 bytes
assets/read_benchmark.png | Bin 1496810 -> 0 bytes
assets/sample.png | Bin 1371083 -> 1443751 bytes
assets/server.png | Bin 1159915 -> 1176428 bytes
assets/web_ui.png | Bin 158894 -> 145001 bytes
assets/write_benchmark.png | Bin 1576176 -> 0 bytes
10 files changed, 59 insertions(+), 30 deletions(-)
diff --git a/README.md b/README.md
index 6be9677c..cf5fe490 100644
--- a/README.md
+++ b/README.md
@@ -10,11 +10,11 @@
[](https://crates.io/crates/iggy)
[](https://crates.io/crates/iggy)
[](https://docs.rs/iggy)
- [](https://github.com/iggy-rs/iggy/actions/workflows/test.yml)
- [](https://coveralls.io/github/iggy-rs/iggy?branch=master)
- [](https://deps.rs/repo/github/iggy-rs/iggy)
- [](https://twitter.com/iggy_rs_)
- [](https://discord.gg/C5Sux5NcRa)
+ [](https://github.com/apache/iggy/actions/workflows/test.yml)
+ [](https://coveralls.io/github/apache/iggy?branch=master)
+ [](https://deps.rs/repo/github/apache/iggy)
+ [](https://twitter.com/ApacheIggy)
+ [](https://discord.gg/C5Sux5NcRa)
</div>
@@ -28,9 +28,7 @@
---
-**Iggy** is the persistent message streaming platform written in Rust, supporting [QUIC](https://www.chromium.org/quic/), TCP (custom binary specification) and HTTP (regular REST API) transport protocols. Currently, running as a single server, it allows creating streams, topics, partitions and segments, and send/receive messages to/from them. The **messages are stored on disk as an append-only log**, and are persisted between restarts.
-
-The goal of the project is to make a distributed streaming platform (running as a cluster), which will be able to scale horizontally and handle **millions of messages per second** (actually, **it's already very fast**, see the benchmarks below).
+**Iggy** is the persistent message streaming platform written in Rust, supporting [QUIC](https://www.chromium.org/quic/), TCP (custom binary specification) and HTTP (regular REST API) transport protocols, **capable of processing millions of messages per second at the low latency**.
Iggy provides **exceptionally high throughput and performance** while utilizing minimal computing resources.
@@ -38,7 +36,7 @@ This is **not yet another extension** running on top of the existing infrastruct
Iggy is the persistent message streaming log **built from the ground up** using the low lvl I/O for speed and efficiency.
-The name is an abbreviation for the Italian Greyhound - small yet extremely fast dogs, the best in their class. Just like mine lovely [Fabio & Cookie](https://www.instagram.com/fabio.and.cookie/) ❤️
+The name is an abbreviation for the Italian Greyhound - small yet extremely fast dogs, the best in their class. See the lovely [Fabio & Cookie](https://www.instagram.com/fabio.and.cookie/) ❤️
---
@@ -53,6 +51,7 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast
- Fully operational RESTful API which can be optionally enabled
- Available client SDK in multiple languages
- **Works directly with the binary data** (lack of enforced schema and serialization/deserialization)
+- Custom **Zero-copy (de)serialization**, which greatly improves the performance and reduces memory usage.
- Configurable server features (e.g. caching, segment size, data flush interval, transport protocols etc.)
- Possibility of storing the **consumer offsets** on the server
- Multiple ways of polling the messages:
@@ -73,23 +72,36 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast
- Built-in **CLI** to manage the streaming server installable via `cargo install iggy-cli`
- Built-in **benchmarking app** to test the performance
- **Single binary deployment** (no external dependencies)
-- Running as a single node (no cluster support yet)
+- Running as a single node (clustering based on Viewstamped Replication will be implemented in the near future)
+
+*Server*
+
+*Files structure*
+
+---
+
+## Architecture
+
+This is the high-level architecture of the Iggy message streaming server, where extremely high performance and ultra low and stable tail latencies are the primary goals. The server is designed to handle high throughput and very low latency (submillisecond tail latencies), making it suitable for real-time applications. For more details, please refer to the [architecture documentation](https://iggy.apache.org/docs/introduction/architecture).
+
+*Architecture*
+
---
## Roadmap
-- Low level optimizations such zero-copy deserialization with [rkyv](https://github.com/rkyv/rkyv) (WiP)
-- Advanced Web UI (WiP)
-- Shared-nothing design and io_uring support (on experimental branch)
-- Clustering & data replication (on sandbox project)
-- Developer friendly SDK supporting multiple languages
-- Plugins & extensions support
+- **Shared-nothing** design and **io_uring** support (PoC on experimental branch, WiP on the main branch)
+- **Clustering** & data replication based on **[VSR](https://pmg.csail.mit.edu/papers/vr-revisited.pdf)** (on sandbox project using Raft, will be implemented after shared-nothing design is completed)
+- Plugins & extensions support (design and PoC as discussed [here](https://github.com/apache/iggy/discussions/1670), WiP)
+
---
## Supported languages SDK (work in progress)
+We're in the process of migrating all the remaining SDKs and other tooling from [iggy-rs](https://github.com/iggy-rs/) organization to this monorepo (WiP).
+
- [Rust](https://crates.io/crates/iggy)
- [C#](https://github.com/iggy-rs/iggy-dotnet-client)
- [Go](https://github.com/iggy-rs/iggy-go-client)
@@ -111,7 +123,7 @@ Iggy CLI can be installed with `cargo install iggy-cli` and then simply accessed
## Web UI
-There's an ongoing effort to build the administrative web UI for the server, which will allow to manage the streams, topics, partitions, messages and so on. Check the [Web UI repository](https://github.com/iggy-rs/iggy-web-ui)
+There's a dedicated Web UI for the server, which allows managing the streams, topics, partitions, browsing the messages and so on. This is an ongoing effort to build the compressive dashboard for the administrative purposes of the Iggy server. Check the [Web UI repository](https://github.com/iggy-rs/iggy-web-ui). The docker image for Web UI is available [here](https://hub.docker.com/r/iggyrs/iggy-web-ui), and can be fetched via `docker pull iggyrs/iggy-web-ui`.
@@ -119,13 +131,15 @@ There's an ongoing effort to build the administrative web UI for the server, whi
## Docker
+The official images can be found [here](https://hub.docker.com/r/apache/iggy), simply type `docker pull apache/iggy` to pull the image.
+
+Please note that the images tagged as `latest` are based on the official, stable releases, while the `edge` ones are updated directly from latest version of the `master` branch.
+
You can find the `Dockerfile` and `docker-compose` in the root of the repository. To build and start the server, run: `docker compose up`.
Additionally, you can run the `CLI` which is available in the running container, by executing: `docker exec -it iggy-server /iggy`.
-Keep in mind that running the container on the OS other than Linux, where the Docker is running in the VM, might result in the significant performance degradation.
-
-The official images can be found [here](https://hub.docker.com/r/iggyrs/iggy), simply type `docker pull iggyrs/iggy`.
+Keep in mind that running the container on the OS other than Linux, where the Docker is running in the VM, might result in the performance degradation.
---
@@ -135,15 +149,15 @@ The default configuration can be found in `server.toml` file in `configs` direct
The configuration file is loaded from the current working directory, but you can specify the path to the configuration file by setting `IGGY_CONFIG_PATH` environment variable, for example `export IGGY_CONFIG_PATH=configs/server.toml` (or other command depending on OS).
-When config file is not found, the default values from embedded server.toml file are used.
+When config file is not found, the default values from embedded `server.toml` file are used.
-For the detailed documentation of the configuration file, please refer to the [configuration](https://iggy.apache.org/docs//server/configuration) section.
+For the detailed documentation of the configuration file, please refer to the [configuration](https://iggy.apache.org/docs/server/configuration) section.
---
## Quick start
-Build the project (the longer compilation time is due to [LTO](https://doc.rust-lang.org/rustc/linker-plugin-lto.html) enabled in release [profile](https://github.com/spetz/iggy/blob/master/Cargo.toml#L2)):
+Build the project (the longer compilation time is due to [LTO](https://doc.rust-lang.org/rustc/linker-plugin-lto.html) enabled in release [profile](https://github.com/apache/iggy/blob/master/Cargo.toml#L2)):
`cargo build`
@@ -155,6 +169,10 @@ Start the server:
`cargo r --bin iggy-server`
+To quickly generate the sample data:
+
+`cargo r --bin data-seeder-tool`
+
*Please note that all commands below are using `iggy` binary, which is part of release (`cli` sub-crate).*
Create a stream with name `dev` (numerical ID will be assigned by server automatically) using default credentials and `tcp` transport (available transports: `quic`, `tcp`, `http`, default `tcp`):
@@ -195,14 +213,10 @@ Poll messages by a regular consumer with ID 1 from the stream `dev` for topic `s
Finally, restart the server to see it is able to load the persisted data.
-The HTTP API endpoints can be found in [server.http](https://github.com/spetz/iggy/blob/master/server/server.http) file, which can be used with [REST Client](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) extension for VS Code.
+The HTTP API endpoints can be found in [server.http](https://github.com/apache/iggy/blob/master/server/server.http) file, which can be used with [REST Client](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) extension for VS Code.
To see the detailed logs from the CLI/server, run it with `RUST_LOG=trace` environment variable. See images below:
-*Files structure*
-
-*Server*
-
---
## Examples
@@ -211,7 +225,7 @@ You can find the sample consumer & producer applications under `examples` direct
To run the example, first start the server with `cargo r --bin iggy-server` and then run the producer and consumer apps with `cargo r --example message-envelope-producer` and `cargo r --example message-envelope-consumer` respectively.
-You might start multiple producers and consumers at the same time to see how the messages are being handled across multiple clients. Check the [Args](https://github.com/spetz/iggy/blob/master/examples/src/shared/args.rs) struct to see the available options, such as the transport protocol, stream, topic, partition, consumer ID, message size etc.
+You might start multiple producers and consumers at the same time to see how the messages are being handled across multiple clients. Check the [Args](https://github.com/apache/iggy/blob/master/examples/src/shared/args.rs) struct to see the available options, such as the transport protocol, stream, topic, partition, consumer ID, message size etc.
By default, the consumer will poll the messages using the `next` available offset with auto commit enabled, to store its offset on the server. With this approach, you can easily achieve *at-most-once* delivery.
@@ -271,6 +285,16 @@ while let Some(message) = consumer.next().await {
## Benchmarks
+**Benchmarks should be the first-class citizens**. We believe that performance is crucial for any system, and we strive to provide the best possible performance for our users. Please check, why we believe that the **[transpart benchmarking](https://iggy.apache.org/blogs/2025/02/17/transparent-benchmarks)** is so important.
+
+We've also built the **[benchmarking platform](https://benchmarks.iggy.rs)** where anyone can upload the benchmarks and compare the results with others. This is the another open-source project available [here](https://github.com/iggy-rs/iggy-bench-dashboard).
+
+*Bench Platform*
+
+For the benchmarking purposes, we've developed the dedicated **iggy-bench** tool, which is a part of the **iggy** project. It is a command-line tool that allows you to run the variety of fully customizable benchmarks.
+
+*Bench CLI*
+
To benchmark the project, first build the project in release mode:
```bash
@@ -322,8 +346,13 @@ Then, run the benchmarking app with the desired options:
```
These benchmarks would start the server with the default configuration, create a stream, topic and partition, and then send or poll the messages. The default configuration is optimized for the best performance, so you might want to tweak it for your needs. If you need more options, please refer to `iggy-bench` subcommands `help` and `examples`.
+
For example, to run the benchmark for the already started server, provide the additional argument `--server-address 0.0.0.0:8090`.
-Depending on the hardware, transport protocol (`quic`, `tcp` or `http`) and payload size (`messages-per-batch * message-size`) you might expect **over 5000 MB/s (e.g. 5M of 1 KB msg/sec) throughput for writes and reads**. These results have been achieved on Ryzen 9 7950X with 64 GB RAM and gen 4 NVMe SSD.
+Depending on the hardware, transport protocol (`quic`, `tcp` or `http`) and payload size (`messages-per-batch * message-size`) you might expect **over 5000 MB/s (e.g. 5M of 1 KB msg/sec) throughput for writes and reads**.
+
+**Iggy is already capable of processing millions of messages per second at the microseconds range for p99+ latency**, and with the upcoming optimizations related to the io_uring support along with the shared-nothing design, it will only get better.
+
+Please refer to the mentioned [benchmarking platform](https://benchmarks.iggy.rs) where you can browse the results achieved on the different hardware configurations, using the different Iggy server versions.
---
diff --git a/assets/bench.png b/assets/bench.png
new file mode 100644
index 00000000..5bf0eacc
Binary files /dev/null and b/assets/bench.png differ
diff --git a/assets/cli.png b/assets/cli.png
index 3e4d63a7..43acc469 100644
Binary files a/assets/cli.png and b/assets/cli.png differ
diff --git a/assets/files_structure.png b/assets/files_structure.png
index 6b5c8d6f..21fd3a2f 100644
Binary files a/assets/files_structure.png and b/assets/files_structure.png differ
diff --git a/assets/iggy_architecture.png b/assets/iggy_architecture.png
new file mode 100644
index 00000000..0d1c885f
Binary files /dev/null and b/assets/iggy_architecture.png differ
diff --git a/assets/read_benchmark.png b/assets/read_benchmark.png
deleted file mode 100644
index 3bed66bf..00000000
Binary files a/assets/read_benchmark.png and /dev/null differ
diff --git a/assets/sample.png b/assets/sample.png
index c6ffb2d6..b9f5349d 100644
Binary files a/assets/sample.png and b/assets/sample.png differ
diff --git a/assets/server.png b/assets/server.png
index 4b416ec4..02e27e0e 100644
Binary files a/assets/server.png and b/assets/server.png differ
diff --git a/assets/web_ui.png b/assets/web_ui.png
index 89c2d769..1fa0d7db 100644
Binary files a/assets/web_ui.png and b/assets/web_ui.png differ
diff --git a/assets/write_benchmark.png b/assets/write_benchmark.png
deleted file mode 100644
index bef931f7..00000000
Binary files a/assets/write_benchmark.png and /dev/null differ