You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iggy.apache.org by "jiengup (via GitHub)" <gi...@apache.org> on 2026/07/26 08:38:37 UTC

Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

GitHub user jiengup created a discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

## Problem

Issue raised by #3735

When the Iggy server rejects a request, the binary protocol response carries only a
4-byte discriminant (e.g., `3007` for `PartitionNotFound`). The actual field values
(partition ID, topic ID, stream ID) are discarded before hitting the wire:

```
Server builds:    PartitionNotFound(5, topic_42, stream_1)
send_error_response → [3007_u32_le] [0x00000000]    ← empty payload
Client rebuilds:  PartitionNotFound(0, Identifier(0), Identifier(0))
```

Due to `&[]` payload in `send_error_response<T>` in `core/server/src/sender/mod.rs` here: https://github.com/apache/iggy/blob/d9635b6d71dacbe6be1702a0723df50f5574b789/core/server/src/sender/mod.rs#L199-L207

Every data-bearing `IggyError` variant (`StreamIdNotFound`, `TopicIdNotFound`,
`InvalidOffset`, `ClientNotFound`, …) arrives with zeroed or empty fields. Two
genuinely different requests produce byte-identical errors. The HTTP transport is
unaffected (it serializes `error.to_string()` as JSON `reason`), proving the data
exists and is merely not transported.

## Current SDK behavior

Each SDK handles the `[status:4][length:4][body:N]` response frame independently,
with widely varying treatment of the error body:

| SDK | Wire parsing | Reads body on error? | Body treatment | Exception produced |
|---|---|---|---|---|
| **C++** | Rust FFI | Yes (inside Rust) | `IggyError` → formatted string | `std::runtime_error` |
| **C#** | Own (raw socket) | Yes | UTF-8 decoded → exception message | `IggyInvalidStatusCodeException` |
| **Go** | Own (raw socket) | Reads and **discards** | Unused | Typed `IggyError` structs (code only) |
| **Java** | Own (Netty) | Yes | UTF-8 decoded → exception reason | `IggyServerException` + typed subclasses |
| **Node.js** | Own (raw socket) | Frame includes body, but **ignores** body content | Unused | Generic `Error` |
| **PHP** | Rust FFI | Yes (inside Rust) | `IggyError` → PHP exception | Typed PHP exception hierarchy |
| **Python** | Rust FFI | Yes (inside Rust) | `IggyError` → `RuntimeError(e.to_string())` | `RuntimeError` (undifferentiated) |

Three SDKS do their own parsing (C#, Go, Java). Four delegate to Rust (C++, Python,
PHP, Node via different approaches). **The body contents are currently unused in
practice because the server sends an empty payload.**

## Proposed approaches

### Option A — String payload (lightweight)

The server sends `error.to_string()` as the response body:

```
Server:   PartitionNotFound(5, topic_42, stream_1)
  └─ body = error.to_string()
     = "Partition with ID: 5 for topic with ID: 42 for stream with ID: 1 was not found."

Wire:     [status=3007][len=80]["Partition with ID: 5 for topic..."]

Client:   reads body → decodes as UTF-8 → displays to caller
           error type is determined by the status code alone (e.g., 3007 → PartitionNotFound)
```

**Pros**
- Minimal SDK changes: C# and Java already decode the body as UTF-8 string today.
  Go and Node.js need only to consume or surface the body bytes.
- The error code in the header preserves the **semantic type** of the error (callers can still inspect the status code to differentiate `PartitionNotFound` from `InvalidOffset`, etc.).
- No wire-format specification doc needed — the contract is "the body is a UTF-8 encoded human-readable error description."

**Cons**
- Callers cannot reconstruct a typed error with correct field values (e.g., a structured `PartitionNotFound` object with `partition_id=5`) from a flat string. For most use cases (logging, debug output), the human-readable string is sufficient.

---

### Option B — Structured serialization (full fidelity)

Define a wire-format specification that carries the error's fields as structured binary data, enabling callers to reconstruct the original typed error with every field populated.

Two sub-options:

#### B1. Custom specification

Define a simple binary layout (e.g., field-order matches the variant's tuple fields, integers as LE bytes, identifiers as `[kind:u8][len:u8][bytes]`). This is what was prototyped in #3751.

**Pros**: No external dependency. Full control over the wire format.
**Cons**: Must be manually reimplemented in every SDK that does its own wire parsing (C#, Go, Java, Node.js). Requires a specification document. Every new error variant must be added to the spec and each implementation.

#### B2. Standard serialization framework

Use a cross-language framework such as **protobuf** (already used in the connectors module) or [**Apache Fury**](https://github.com/apache/fory) to serialize error payloads.

**Pros**: Code generation for all target languages. Schema serves as documentation. No hand-written parser maintenance.
**Cons**: Introduces a dependency to every SDK. Serialization overhead for small error payloads (~20–80 bytes). Protobuf/Avro schema must be kept in sync with `IggyError` variants.

---

## Questions for discussion

1. **Is full structured error reconstruction necessary?** In practice, are callers inspecting error field values (e.g., the rejected partition ID) in application logic, or is the human-readable message sufficient?

2. **If structured, B1 (custom) or B2 (framework)?** Protobuf is already a transitive dependency via connectors. Would adding it to the error path be acceptable, or is a lightweight custom spec preferred?

3. **Backward compatibility.** Option A requires no changes to Java and C# (they already decode UTF-8 bodies). Option B requires changes to ALL self-parsing SDKs (C#, Go, Java, Node.js). Is the additional fidelity worth the per-SDK implementation cost?


GitHub link: https://github.com/apache/iggy/discussions/3752

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "jiengup (via GitHub)" <gi...@apache.org>.
GitHub user jiengup added a comment to the discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

A sample protobuf message definition will look like this:

```protobuf
syntax = "proto3";

message ErrorPayload {
  uint32 schema_version = 1;
  string message = 2;

  oneof details {
    PartitionNotFoundDetails partition_not_found = 10;
    ResourceDetails resource = 11;
    ChecksumDetails checksum = 12;
  }
}

message PartitionNotFoundDetails {
  uint64 partition_id = 1;
  Identifier topic_id = 2;
  Identifier stream_id = 3;
}

message Identifier {
  oneof value {
    uint32 id = 1;
    string name = 2;
  }
}
```

GitHub link: https://github.com/apache/iggy/discussions/3752#discussioncomment-17785410

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "jiengup (via GitHub)" <gi...@apache.org>.
GitHub user jiengup added a comment to the discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Thanks for your context. I think flatbuffer is also a good choice. It's be used in some famous projects such as Apache Arrow as I know.

GitHub link: https://github.com/apache/iggy/discussions/3752#discussioncomment-17785939

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "jiengup (via GitHub)" <gi...@apache.org>.
GitHub user jiengup added a comment to the discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

After discussion with the core team, it has been decided that we will postpone the error protocol implementation until the VSR cluster feature is stable.

GitHub link: https://github.com/apache/iggy/discussions/3752#discussioncomment-17824466

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "lukaszzborek (via GitHub)" <gi...@apache.org>.
GitHub user lukaszzborek added a comment to the discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Some time ago, there was idea for using flatbuffers https://github.com/apache/iggy/issues/2524. 

GitHub link: https://github.com/apache/iggy/discussions/3752#discussioncomment-17785483

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "lukaszzborek (via GitHub)" <gi...@apache.org>.
GitHub user lukaszzborek added a comment to the discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Few quick thoughts:
Some time ago, there was idea for using flatbuffers https://github.com/apache/iggy/issues/2524 to autogenerate errors for all languages.
I think that at this stage, we shouldn't worry about backward compatibility just yet.

In my opinion, we should go to option B.  I'm not sure if we need Apache.Fury (I don't know it, I need to check it out). Server also has to send every field value and rendered message from server (like in http). This means we don't need to store message templates in every sdk, and we also have way to get all parts of field values like topic_id, stream_id (like in your example protobuf).
Also propably in C#/Java/Go we want to have some sync enum/typed-error list for error codes, instead of just numbers (like now in c#). It can be generated like in go (or my old pr 😅), but I think it should be also topic of that discussion to cover that


GitHub link: https://github.com/apache/iggy/discussions/3752#discussioncomment-17785858

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "jiengup (via GitHub)" <gi...@apache.org>.
GitHub user jiengup edited a discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

## Problem

Issue raised by #3735

When the Iggy server rejects a request, the binary protocol response carries only a
4-byte discriminant (e.g., `3007` for `PartitionNotFound`). The actual field values
(partition ID, topic ID, stream ID) are discarded before hitting the wire:

```
Server builds:    PartitionNotFound(5, topic_42, stream_1)
send_error_response → [3007_u32_le] [0x00000000]    ← empty payload
Client rebuilds:  PartitionNotFound(0, Identifier(0), Identifier(0))
```

Due to `&[]` payload in `send_error_response<T>` in `core/server/src/sender/mod.rs` here: https://github.com/apache/iggy/blob/d9635b6d71dacbe6be1702a0723df50f5574b789/core/server/src/sender/mod.rs#L199-L207

Every data-bearing `IggyError` variant (`StreamIdNotFound`, `TopicIdNotFound`,
`InvalidOffset`, `ClientNotFound`, …) arrives with zeroed or empty fields. Two
genuinely different requests produce byte-identical errors. The HTTP transport is
unaffected (it serializes `error.to_string()` as JSON `reason`), proving the data
exists and is merely not transported.

## Current SDK behavior

Each SDK handles the `[status:4][length:4][body:N]` response frame independently,
with widely varying treatment of the error body:

| SDK | Wire parsing | Reads body on error? | Body treatment | Exception produced |
|---|---|---|---|---|
| **C++** | Rust FFI | Yes (inside Rust) | `IggyError` → formatted string | `std::runtime_error` |
| **C#** | Own (raw socket) | Yes | UTF-8 decoded → exception message | `IggyInvalidStatusCodeException` |
| **Go** | Own (raw socket) | Reads and **discards** | Unused | Typed `IggyError` structs (code only) |
| **Java** | Own (Netty) | Yes | UTF-8 decoded → exception reason | `IggyServerException` + typed subclasses |
| **Node.js** | Own (raw socket) | Frame includes body, but **ignores** body content | Unused | Generic `Error` |
| **PHP** | Rust FFI | Yes (inside Rust) | `IggyError` → PHP exception | Typed PHP exception hierarchy |
| **Python** | Rust FFI | Yes (inside Rust) | `IggyError` → `RuntimeError(e.to_string())` | `RuntimeError` (undifferentiated) |

Three SDKS do their own parsing (C#, Go, Java). Four delegate to Rust (C++, Python,
PHP, Node via different approaches). **The body contents are currently unused in
practice because the server sends an empty payload.**

## Proposed approaches

### Option A — String payload (lightweight)

The server sends `error.to_string()` as the response body:

```
Server:   PartitionNotFound(5, topic_42, stream_1)
  └─ body = error.to_string()
     = "Partition with ID: 5 for topic with ID: 42 for stream with ID: 1 was not found."

Wire:     [status=3007][len=80]["Partition with ID: 5 for topic..."]

Client:   reads body → decodes as UTF-8 → displays to caller
           error type is determined by the status code alone (e.g., 3007 → PartitionNotFound)
```

**Pros**
- Minimal SDK changes: C# and Java already decode the body as UTF-8 string today.
  Go and Node.js need only to consume or surface the body bytes.
- The error code in the header preserves the **semantic type** of the error (callers can still inspect the status code to differentiate `PartitionNotFound` from `InvalidOffset`, etc.).
- No wire-format specification doc needed — the contract is "the body is a UTF-8 encoded human-readable error description."

**Cons**
- Callers cannot reconstruct a typed error with correct field values (e.g., a structured `PartitionNotFound` object with `partition_id=5`) from a flat string. For most use cases (logging, debug output), the human-readable string is sufficient.

---

### Option B — Structured serialization (full fidelity)

Define a wire-format specification that carries the error's fields as structured binary data, enabling callers to reconstruct the original typed error with every field populated.

Two sub-options:

#### B1. Custom specification

Define a simple binary layout (e.g., field-order matches the variant's tuple fields, integers as LE bytes, identifiers as `[kind:u8][len:u8][bytes]`). This is what was prototyped in #3751.

**Pros**: No external dependency. Full control over the wire format.
**Cons**: Must be manually reimplemented in every SDK that does its own wire parsing (C#, Go, Java, Node.js). Requires a specification document. Every new error variant must be added to the spec and each implementation.

#### B2. Standard serialization framework

Use a cross-language framework such as **protobuf** (already used in the connectors module) or [**Apache Fury**](https://github.com/apache/fory) to serialize error payloads.

**Pros**: Code generation for all target languages. Schema serves as documentation. No hand-written parser maintenance.
**Cons**: Introduces a dependency to every SDK. Serialization overhead for small error payloads (~20–80 bytes). Protobuf/Avro schema must be kept in sync with `IggyError` variants.

---

## Questions for discussion

1. **Is full structured error reconstruction necessary?** In practice, are callers inspecting error field values (e.g., the rejected partition ID) in application logic, or is the human-readable message sufficient?

2. **If structured, B1 (custom) or B2 (framework)?** Protobuf is already a transitive dependency via connectors. Would adding it to the error path be acceptable, or is a lightweight custom spec preferred?

3. **Backward compatibility.** Option A requires no changes to Java and C# (they already decode UTF-8 bodies). Option B requires changes to ALL self-parsing SDKs (C#, Go, Java, Node.js). Is the additional fidelity worth the per-SDK implementation cost?

## Related Issues and PRs
- https://github.com/apache/iggy/issues/2524
- https://github.com/apache/iggy/issues/2519


GitHub link: https://github.com/apache/iggy/discussions/3752

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "jiengup (via GitHub)" <gi...@apache.org>.
GitHub user jiengup edited a comment on the discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

After discussion with the core team, it has been decided that we will postpone the error protocol implementation until the VSR cluster feature is stable. 

This proposal can be used for tracking and further discussion.

GitHub link: https://github.com/apache/iggy/discussions/3752#discussioncomment-17824466

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
GitHub user hubcio added a comment to the discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

I will prepare design doc and post it here. I actually have something in mind, to finally get rid of big nasty IggyError and have per-command individual errors + mechanism that allows cross-sdk maintainability of it. 

GitHub link: https://github.com/apache/iggy/discussions/3752#discussioncomment-17824545

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org


Re: Proposal: Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Posted by "lukaszzborek (via GitHub)" <gi...@apache.org>.
GitHub user lukaszzborek deleted a comment on the discussion: Proposal:  Carrying error detail over the TCP/QUIC/WebSocket binary protocol

Some time ago, there was idea for using flatbuffers https://github.com/apache/iggy/issues/2524. 

GitHub link: https://github.com/apache/iggy/discussions/3752#discussioncomment-17785483

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org