You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by hg...@apache.org on 2025/06/19 09:51:30 UTC
(iggy) 01/01: chore(ci): fix examples and documentation issues
This is an automated email from the ASF dual-hosted git repository.
hgruszecki pushed a commit to branch test-examples
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 587bd2cb3a4cc95d3433eb895b6d3656cab38425
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Sat Feb 22 12:25:03 2025 +0100
chore(ci): fix examples and documentation issues
This commit fixes a producer initialization bug where the
initialized flag was incorrectly set to true, preventing proper topic
creation during producer setup. The multi-tenant consumer example now
includes proper termination logic to prevent infinite loops. A bug in
the producer create_topic method that caused empty topic names has
been resolved by using the extracted name variable instead of
topic_name.
The README navigation links have been centered for better formatting.
Missing send_messages permissions were added to multi-tenant users to
enable proper message sending. Documentation warnings were fixed by
escaping HTML tags and brackets in rustdoc comments. The CI workflow
now includes cargo doc validation to catch documentation issues.
---
.github/workflows/ci-check-rust.yml | 5 +-
.markdownlint.json | 6 ++
README.md | 97 +++++++++++++++++++---
core/cli/src/args/user.rs | 4 +-
core/common/src/types/args/mod.rs | 2 +-
core/common/src/types/identifier/mod.rs | 2 +-
core/common/src/types/message/messages_batch.rs | 4 +-
core/examples/src/multi-tenant/consumer/main.rs | 17 +++-
.../tests/cli/general/test_help_command.rs | 2 +-
.../tests/cli/user/test_user_create_command.rs | 2 +-
.../cli/user/test_user_permissions_command.rs | 2 +-
core/sdk/src/clients/producer.rs | 2 +-
scripts/run-examples-from-readme.sh | 66 +++++++++++----
13 files changed, 169 insertions(+), 42 deletions(-)
diff --git a/.github/workflows/ci-check-rust.yml b/.github/workflows/ci-check-rust.yml
index ac48d57d..202a48c5 100644
--- a/.github/workflows/ci-check-rust.yml
+++ b/.github/workflows/ci-check-rust.yml
@@ -145,6 +145,9 @@ jobs:
- name: Run cargo test (doc)
run: |
cargo test --doc
+ - name: Run cargo doc
+ run: |
+ cargo doc --no-deps --all-features --quiet
unused_dependencies:
name: cargo machete
@@ -162,4 +165,4 @@ jobs:
tool: cargo-machete
- name: Run cargo machete
run: |
- cargo machete --with-metadata
\ No newline at end of file
+ cargo machete --with-metadata
diff --git a/.markdownlint.json b/.markdownlint.json
new file mode 100644
index 00000000..1a958ff5
--- /dev/null
+++ b/.markdownlint.json
@@ -0,0 +1,6 @@
+{
+ "MD033": false,
+ "MD013": {
+ "line_length": 500
+ }
+}
diff --git a/README.md b/README.md
index a254c4f4..909913a2 100644
--- a/README.md
+++ b/README.md
@@ -1,16 +1,14 @@
# Apache Iggy (Incubating)
-<div style="display: flex; flex-wrap: wrap; justify-content: center; align-items: center; text-align: center;">
+<div align="center">
- [Website](https://iggy.apache.org) | [Getting started](https://iggy.apache.org/docs/introduction/getting-started/) | [Documentation](https://iggy.apache.org/docs/) | [Blog](https://iggy.apache.org/blogs/) | [Discord](https://discord.gg/C5Sux5NcRa) | [Crates](https://crates.io/crates/iggy)
+[Website](https://iggy.apache.org) | [Getting started](https://iggy.apache.org/docs/introduction/getting-started/) | [Documentation](https://iggy.apache.org/docs/) | [Blog](https://iggy.apache.org/blogs/) | [Discord](https://discord.gg/C5Sux5NcRa) | [Crates](https://crates.io/crates/iggy)
</div>
<div style="display: flex; flex-wrap: wrap; justify-content: center; align-items: center; text-align: center;">
[](https://crates.io/crates/iggy)
[](https://crates.io/crates/iggy)
- [](https://docs.rs/iggy)
- [](https://github.com/apache/iggy/actions/workflows/test.yml)
[](https://coveralls.io/github/iggy-rs/iggy?branch=master)
[](https://deps.rs/repo/github/apache/iggy)
[](https://twitter.com/ApacheIggy)
@@ -63,7 +61,7 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast
- **Consumer groups** providing the message ordering and horizontal scaling across the connected clients
- **Message expiry** with auto deletion based on the configurable **retention policy**
- Additional features such as **server side message deduplication**
-- **Multi-tenant** support via abstraction of **streams** whch group **topics**
+- **Multi-tenant** support via abstraction of **streams** which group **topics**
- **TLS** support for all transport protocols (TCP, QUIC, HTTPS)
- **[Connectors](https://github.com/apache/iggy/tree/master/core/connectors)** - sinks, sources and data transformations based on the **custom Rust plugins**
- Optional server-side as well as client-side **data encryption** using AES-256-GCM
@@ -79,12 +77,11 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast
-
---
## Architecture
-This is the high-level architecture of the Iggy message streaming server, where extremely high performance and ultra low and stable tail latencies are the primary goals. The server is designed to handle high throughput and very low latency (submillisecond tail latencies), making it suitable for real-time applications. For more details, please refer to the [documentation](https://iggy.apache.org/docs/introduction/architecture).
+This is the high-level architecture of the Iggy message streaming server, where extremely high performance and ultra low and stable tail latencies are the primary goals. The server is designed to handle high throughput and very low latency (sub-millisecond tail latencies), making it suitable for real-time applications. For more details, please refer to the [documentation](https://iggy.apache.org/docs/introduction/architecture).
@@ -116,7 +113,6 @@ The recent improvements based on the zero-copy (de)serialization, along with upd
- C++
- Elixir
-
Almost all the tooling has been migrated from [iggy-rs](https://github.com/iggy-rs/) organization to this monorepo. There's no an ongoing work to support the zero-copy (de)serialization for all the SDKs.
---
@@ -131,7 +127,7 @@ Iggy CLI can be installed with `cargo install iggy-cli` and then simply accessed
## Web UI
-There's a dedicated Web UI for the server, which allows managing the streams, topics, partitions, browsing the messages and so on. This is an ongoing effort to build the compressive dashboard for the administrative purposes of the Iggy server. Check the Web UI in the `/web` directory. The docker image for Web UI is available [here](https://hub.docker.com/r/iggyrs/iggy-web-ui), and can be fetched via `docker pull iggyrs/iggy-web-ui`.
+There's a dedicated Web UI for the server, which allows managing the streams, topics, partitions, browsing the messages and so on. This is an ongoing effort to build the compressive dashboard for the administrative purposes of the Iggy server. Check the Web UI in the `/web` directory. The [docker image for Web UI](https://hub.docker.com/r/iggyrs/iggy-web-ui) is available, and can be fetched via `docker pull iggyrs/iggy-web-ui`.
@@ -139,7 +135,7 @@ There's a dedicated Web UI for the server, which allows managing the streams, to
## Docker
-The official images can be found [here](https://hub.docker.com/r/apache/iggy), simply type `docker pull apache/iggy` to pull the image.
+The official images can be found [in Docker Hub](https://hub.docker.com/r/apache/iggy), simply type `docker pull apache/iggy` to pull the image.
Please note that the images tagged as `latest` are based on the official, stable releases, while the `edge` ones are updated directly from latest version of the `master` branch.
@@ -231,7 +227,7 @@ To see the detailed logs from the CLI/server, run it with `RUST_LOG=trace` envir
You can find the sample consumer & producer applications under `examples` directory. The purpose of these apps is to showcase the usage of the client SDK. To find out more about building the applications, please refer to the [getting started](https://iggy.apache.org/docs/introduction/getting-started) guide.
-To run the example, first start the server with `cargo r --bin iggy-server` and then run the producer and consumer apps with `cargo r --example message-envelope-producer` and `cargo r --example message-envelope-consumer` respectively.
+To run the example, first start the server with `cargo run --bin iggy-server` and then run the producer and consumer apps with `cargo run --example message-envelope-producer` and `cargo run --example message-envelope-consumer` respectively.
You might start multiple producers and consumers at the same time to see how the messages are being handled across multiple clients. Check the [Args](https://github.com/apache/iggy/blob/master/core/examples/src/shared/args.rs) struct to see the available options, such as the transport protocol, stream, topic, partition, consumer ID, message size etc.
@@ -239,6 +235,83 @@ By default, the consumer will poll the messages using the `next` available offse
+### Getting Started
+
+Basic example showing how to connect to Iggy and perform fundamental operations. Perfect for newcomers.
+
+```bash
+# Run the producer
+cargo run --example getting-started-producer
+
+# Run the consumer
+cargo run --example getting-started-consumer
+```
+
+### Basic Usage
+
+Demonstrates core functionality with detailed comments explaining each operation.
+
+```bash
+cargo run --example basic-producer
+cargo run --example basic-consumer
+```
+
+### Message Headers
+
+Shows how to work with message headers for metadata management.
+
+```bash
+# Producer adds custom headers to messages
+cargo run --example message-headers-producer
+
+# Consumer reads and processes headers
+cargo run --example message-headers-consumer
+```
+
+### Message Envelope
+
+Demonstrates working with message envelopes for advanced message handling.
+
+```bash
+cargo run --example message-envelope-producer
+cargo run --example message-envelope-consumer
+```
+
+### Multi-tenant Setup
+
+Example of implementing multi-tenancy using streams for isolation.
+
+```bash
+# Start the multi-tenant producer
+cargo run --example multi-tenant-producer
+
+# Start the multi-tenant consumer
+cargo run --example multi-tenant-consumer
+```
+
+### Stream Builder
+
+Shows how to programmatically create and configure streams:
+
+- Setting up partitions
+- Configuring retention policies
+- Managing message expiry
+- Setting up consumer groups
+
+```bash
+cargo run --example stream-builder
+```
+
+### Running Examples
+
+All examples can be run directly from the repository. Make sure to:
+
+1. Start the Iggy server first: `cargo run --bin iggy-server`
+2. Run the examples using the commands shown above
+3. Check the example source code for detailed comments and explanations
+
+The examples are also tested automatically using the `scripts/run-examples-from-readme.sh` script, which ensures they stay up-to-date and working.
+
---
## SDK
@@ -296,7 +369,7 @@ while let Some(message) = consumer.next().await {
**Benchmarks should be the first-class citizens**. We believe that performance is crucial for any system, and we strive to provide the best possible performance for our users. Please check, why we believe that the **[transparent
benchmarking](https://iggy.apache.org/blogs/2025/02/17/transparent-benchmarks)** is so important.
-We've also built the **[benchmarking platform](https://benchmarks.iggy.rs)** where anyone can upload the benchmarks and compare the results with others. This is the another open-source project available [here](https://github.com/iggy-rs/iggy-bench-dashboard).
+We've also built the **[benchmarking platform](https://benchmarks.iggy.rs)** where anyone can upload the benchmarks and compare the results with others. Source code for the platform is available in the `core/bench/dashboard` directory.
diff --git a/core/cli/src/args/user.rs b/core/cli/src/args/user.rs
index ea51533f..40c8118c 100644
--- a/core/cli/src/args/user.rs
+++ b/core/cli/src/args/user.rs
@@ -161,7 +161,7 @@ pub(crate) struct UserCreateArgs {
/// Available topic permissions: manage_topic / m_top, read_topic / r_top, poll_messages / p_msg,
/// send_messages / s_msg.
///
- /// Permissions format: STREAM_ID[:STREAM_PERMISSIONS][#TOPIC_ID[:TOPIC_PERMISSIONS]]
+ /// Permissions format: STREAM_ID\[:STREAM_PERMISSIONS\]\[#TOPIC_ID\[:TOPIC_PERMISSIONS\]\]
///
/// Examples:
/// iggy user create guest guest -s 1:manage_topics,read_topics
@@ -292,7 +292,7 @@ pub(crate) struct UserPermissionsArgs {
/// Available topic permissions: manage_topic / m_top, read_topic / r_top, poll_messages / p_msg,
/// send_messages / s_msg.
///
- /// Permissions format: STREAM_ID[:STREAM_PERMISSIONS][#TOPIC_ID[:TOPIC_PERMISSIONS]]
+ /// Permissions format: STREAM_ID\[:STREAM_PERMISSIONS\]\[#TOPIC_ID\[:TOPIC_PERMISSIONS\]\]
///
/// Examples:
/// iggy user create guest guest -s 1:manage_topics,read_topics
diff --git a/core/common/src/types/args/mod.rs b/core/common/src/types/args/mod.rs
index ab0d6f62..3010faa5 100644
--- a/core/common/src/types/args/mod.rs
+++ b/core/common/src/types/args/mod.rs
@@ -37,7 +37,7 @@ pub struct ArgsOptional {
/// Optional encryption key for the message payload used by the client
///
- /// [default: ]
+ /// (default: none)
#[arg(long)]
#[serde(skip_serializing_if = "Option::is_none")]
pub encryption_key: Option<String>,
diff --git a/core/common/src/types/identifier/mod.rs b/core/common/src/types/identifier/mod.rs
index a6bfcfa8..7a058a4c 100644
--- a/core/common/src/types/identifier/mod.rs
+++ b/core/common/src/types/identifier/mod.rs
@@ -111,7 +111,7 @@ impl Identifier {
self.get_cow_str_value().map(|cow| cow.to_string())
}
- /// Returns the Cow<str> value of the identifier.
+ /// Returns the `Cow<str>` value of the identifier.
pub fn get_cow_str_value(&self) -> Result<Cow<str>, IggyError> {
if self.kind != IdKind::String {
return Err(IggyError::InvalidIdentifier);
diff --git a/core/common/src/types/message/messages_batch.rs b/core/common/src/types/message/messages_batch.rs
index 47d49d8c..0076f7f0 100644
--- a/core/common/src/types/message/messages_batch.rs
+++ b/core/common/src/types/message/messages_batch.rs
@@ -364,7 +364,7 @@ impl From<&[IggyMessage]> for IggyMessagesBatch {
}
}
-/// Converts a reference to Vec<IggyMessage> into an IggyMessagesBatch.
+/// Converts a reference to `Vec<IggyMessage>` into an IggyMessagesBatch.
///
/// This implementation delegates to the slice implementation via `as_slice()`.
/// It's provided for convenience so it's possible to use `&messages` without
@@ -375,7 +375,7 @@ impl From<&Vec<IggyMessage>> for IggyMessagesBatch {
}
}
-/// Converts a Vec<IggyMessage> into an IggyMessagesBatch.
+/// Converts a `Vec<IggyMessage>` into an IggyMessagesBatch.
impl From<Vec<IggyMessage>> for IggyMessagesBatch {
fn from(messages: Vec<IggyMessage>) -> Self {
Self::from(messages.as_slice())
diff --git a/core/examples/src/multi-tenant/consumer/main.rs b/core/examples/src/multi-tenant/consumer/main.rs
index af947f60..267b9991 100644
--- a/core/examples/src/multi-tenant/consumer/main.rs
+++ b/core/examples/src/multi-tenant/consumer/main.rs
@@ -173,7 +173,8 @@ async fn main() -> anyhow::Result<(), Box<dyn Error>> {
));
let mut tasks = Vec::new();
for tenant in tenants.into_iter() {
- let consumer_tasks = start_consumers(tenant.id, tenant.consumers);
+ let consumer_tasks =
+ start_consumers(tenant.id, tenant.consumers, args.message_batches_limit);
tasks.extend(consumer_tasks);
}
@@ -233,14 +234,25 @@ async fn create_user(
Ok(())
}
-fn start_consumers(tenant_id: u32, consumers: Vec<TenantConsumer>) -> Vec<JoinHandle<()>> {
+fn start_consumers(
+ tenant_id: u32,
+ consumers: Vec<TenantConsumer>,
+ message_batches_limit: u64,
+) -> Vec<JoinHandle<()>> {
let mut tasks = Vec::new();
for mut consumer in consumers {
let task = tokio::spawn(async move {
let consumer_id = consumer.id;
let stream = consumer.stream;
let topic = consumer.topic;
+ let mut batches_processed = 0u64;
while let Some(message) = consumer.consumer.next().await {
+ if batches_processed >= message_batches_limit {
+ info!(
+ "Tenant: {tenant_id} consumer: {consumer_id} reached message batches limit: {message_batches_limit}, stopping."
+ );
+ break;
+ }
if let Ok(message) = message {
let current_offset = message.current_offset;
let partition_id = message.partition_id;
@@ -258,6 +270,7 @@ fn start_consumers(tenant_id: u32, consumers: Vec<TenantConsumer>) -> Vec<JoinHa
info!(
"Tenant: {tenant_id} consumer: {consumer_id} received: {payload} from partition: {partition_id}, topic: {topic}, stream: {stream}, at offset: {offset}, current offset: {current_offset}"
);
+ batches_processed += 1;
} else if let Err(error) = message {
error!(
"Error while handling message: {error} by tenant: {tenant_id} consumer: {consumer_id}, topic: {topic}, stream: {stream}"
diff --git a/core/integration/tests/cli/general/test_help_command.rs b/core/integration/tests/cli/general/test_help_command.rs
index 03e07825..8481bc45 100644
--- a/core/integration/tests/cli/general/test_help_command.rs
+++ b/core/integration/tests/cli/general/test_help_command.rs
@@ -61,7 +61,7 @@ Options:
--encryption-key <ENCRYPTION_KEY>
Optional encryption key for the message payload used by the client
{CLAP_INDENT}
- [default: ]
+ (default: none)
--credentials-username <CREDENTIALS_USERNAME>
Optional username for initial login
diff --git a/core/integration/tests/cli/user/test_user_create_command.rs b/core/integration/tests/cli/user/test_user_create_command.rs
index edc07df3..ecae4c47 100644
--- a/core/integration/tests/cli/user/test_user_create_command.rs
+++ b/core/integration/tests/cli/user/test_user_create_command.rs
@@ -347,7 +347,7 @@ Options:
Available topic permissions: manage_topic / m_top, read_topic / r_top, poll_messages / p_msg,
send_messages / s_msg.
{CLAP_INDENT}
- Permissions format: STREAM_ID[:STREAM_PERMISSIONS][#TOPIC_ID[:TOPIC_PERMISSIONS]]
+ Permissions format: STREAM_ID\[:STREAM_PERMISSIONS\]\[#TOPIC_ID\[:TOPIC_PERMISSIONS\]\]
{CLAP_INDENT}
Examples:
iggy user create guest guest -s 1:manage_topics,read_topics
diff --git a/core/integration/tests/cli/user/test_user_permissions_command.rs b/core/integration/tests/cli/user/test_user_permissions_command.rs
index 47da2a5d..c4cf30cb 100644
--- a/core/integration/tests/cli/user/test_user_permissions_command.rs
+++ b/core/integration/tests/cli/user/test_user_permissions_command.rs
@@ -312,7 +312,7 @@ Options:
Available topic permissions: manage_topic / m_top, read_topic / r_top, poll_messages / p_msg,
send_messages / s_msg.
{CLAP_INDENT}
- Permissions format: STREAM_ID[:STREAM_PERMISSIONS][#TOPIC_ID[:TOPIC_PERMISSIONS]]
+ Permissions format: STREAM_ID\[:STREAM_PERMISSIONS\]\[#TOPIC_ID\[:TOPIC_PERMISSIONS\]\]
{CLAP_INDENT}
Examples:
iggy user create guest guest -s 1:manage_topics,read_topics
diff --git a/core/sdk/src/clients/producer.rs b/core/sdk/src/clients/producer.rs
index 43ecf7b0..2c76b4ad 100644
--- a/core/sdk/src/clients/producer.rs
+++ b/core/sdk/src/clients/producer.rs
@@ -450,7 +450,7 @@ impl IggyProducer {
mode: SendMode,
) -> Self {
let core = Arc::new(ProducerCore {
- initialized: AtomicBool::new(true),
+ initialized: AtomicBool::new(false),
client: Arc::new(client),
can_send: Arc::new(AtomicBool::new(true)),
stream_id: Arc::new(stream),
diff --git a/scripts/run-examples-from-readme.sh b/scripts/run-examples-from-readme.sh
index b8b74fdb..fdfdacdf 100755
--- a/scripts/run-examples-from-readme.sh
+++ b/scripts/run-examples-from-readme.sh
@@ -43,15 +43,18 @@ test -d local_data && rm -fr local_data
test -e ${LOG_FILE} && rm ${LOG_FILE}
test -e ${PID_FILE} && rm ${PID_FILE}
+# Build binaries
+echo "Building binaries..."
+cargo build
+
# Run iggy server and let it run in the background
-cargo run --bin iggy-server &> ${LOG_FILE} & echo $! > ${PID_FILE}
+cargo run --bin iggy-server &>${LOG_FILE} &
+echo $! >${PID_FILE}
# Wait until "Iggy server has started" string is present inside iggy-server.log
SERVER_START_TIME=0
-while ! grep -q "Iggy server has started" ${LOG_FILE}
-do
- if [ ${SERVER_START_TIME} -gt ${TIMEOUT} ]
- then
+while ! grep -q "Iggy server has started" ${LOG_FILE}; do
+ if [ ${SERVER_START_TIME} -gt ${TIMEOUT} ]; then
echo "Server did not start within ${TIMEOUT} seconds."
ps fx
cat ${LOG_FILE}
@@ -59,15 +62,14 @@ do
fi
echo "Waiting for Iggy server to start... ${SERVER_START_TIME}"
sleep 1
- ((SERVER_START_TIME+=1))
+ ((SERVER_START_TIME += 1))
done
-# Execute all matching commands from README.md and check if they pass or fail
-while IFS= read -r command
-do
+# Execute all matching CLI commands from README.md and check if they pass or fail
+while IFS= read -r command; do
# Remove backticks from command
command=$(echo "${command}" | tr -d '`')
- echo -e "\e[33mChecking command:\e[0m ${command}"
+ echo -e "\e[33mChecking CLI command:\e[0m ${command}"
echo ""
set +e
@@ -76,26 +78,56 @@ do
set -e
# Stop at first failure
- if [ ${exit_code} -ne 0 ]
- then
+ if [ ${exit_code} -ne 0 ]; then
echo ""
- echo -e "\e[31mCommand failed:\e[0m ${command}"
+ echo -e "\e[31mCLI command failed:\e[0m ${command}"
echo ""
break
fi
done < <(grep -E "^\`cargo r --bin iggy -- " README.md)
+# Execute all example commands from README.md and check if they pass or fail
+while IFS= read -r command; do
+ # Remove backticks and comments from command
+ command=$(echo "${command}" | tr -d '`' | sed 's/^#.*//')
+ # Skip empty lines
+ if [ -z "${command}" ]; then
+ continue
+ fi
+ echo -e "\e[33mChecking example command:\e[0m ${command}"
+ echo ""
+
+ set +e
+ eval "${command}"
+ exit_code=$?
+ set -e
+
+ # Stop at first failure
+ if [ ${exit_code} -ne 0 ]; then
+ echo ""
+ echo -e "\e[31mExample command failed:\e[0m ${command}"
+ echo ""
+ break
+ fi
+ # Add a small delay between examples to avoid potential race conditions
+ sleep 2
+
+done < <(grep -E "^cargo run --example" README.md)
+
# Terminate server
kill -TERM "$(cat ${PID_FILE})"
test -e ${PID_FILE} && rm ${PID_FILE}
# If everything is ok remove log and pid files otherwise cat server log
-if [ "${exit_code}" -eq 0 ]
-then
- test -e ${LOG_FILE} && rm ${LOG_FILE}
+if [ "${exit_code}" -eq 0 ]; then
+ echo "Test passed"
else
- cat ${LOG_FILE}
+ echo "Test failed, see log file:"
+ test -e ${LOG_FILE} && cat ${LOG_FILE}
fi
+test -e ${LOG_FILE} && rm ${LOG_FILE}
+test -e ${PID_FILE} && rm ${PID_FILE}
+
exit "${exit_code}"