You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "ZhiXiao-Lin (via GitHub)" <gi...@apache.org> on 2026/09/30 13:45:29 UTC

[I] Server 0.9.0 intermittently panics on restart boot replay: 'client_id 0 is reserved for internal use' [iggy]

ZhiXiao-Lin opened a new issue, #4361:
URL: https://github.com/apache/iggy/issues/4361

   ## Summary
   
   Restarting a single-node server (`docker restart`, i.e. process restart with the data directory preserved) intermittently kills the shard during boot replay:
   
   ```
   thread 'shard-0' panicked at core/consensus/src/client_table.rs:1129:9:
   client_id 0 is reserved for internal use
   ERROR shard-0 server::boot::threads: message pump died instead of draining (task panicked: client_id 0 is reserved for internal use); committed journal tail may not have flushed
   Error: ShardJoinFailures { failures: [ShardJoinFailure { shard_id: 0, kind: Error(ShardPumpDied { shard_id: 0, reason: "task panicked: client_id 0 is reserved for internal use" }) }] }
   ```
   
   The process exits (1) and cannot boot again with the same data directory. Recreating the container (fresh data) boots clean. The panic is state-dependent: the same workload sometimes restarts cleanly. Reproduced 4 times in one day of integration testing.
   
   ## Repro
   
   Docker (macOS host / Docker Desktop; single-node, TCP only):
   
   ```bash
   docker run -d --name iggy --security-opt seccomp=unconfined -p 5102:5102 \
     -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \
     -e IGGY_TCP_ADDRESS=0.0.0.0:5102 -e IGGY_NODE_ADVERTISED_ADDRESS=127.0.0.1 \
     -e IGGY_SHARDING_CPU_ALLOCATION=1 -e IGGY_SHARDING_PIN_CORES=false \
     apache/iggy:0.9.0
   ```
   
   Then, repeatedly (via the Rust SDK 0.11.0):
   
   1. `login_user("iggy", "iggy")`
   2. create a stream + topic, send a few messages
   3. create/join a consumer group, poll a batch, `store_consumer_offset`
   4. drop the client connection
   5. `docker restart iggy`
   
   Observed: every few cycles, boot replay panics as above and the server stays down. In one reproduction the server died on its own boot with no explicit restart command issued around that time.
   
   ## Suspicion
   
   The persisted client table replays a session whose (reconstructed or replayed) client id collides with the reserved internal id 0 — plausibly a client that was mid-registration when the process stopped, or a session whose id was never durably assigned before the kill. Boot treats the collision as a panic instead of rejecting/ignoring the stale entry, turning a recoverable restart into a hard outage requiring a manual data-dir reset.
   
   ## Impact
   
   Any production single-node deployment that restarts (deploy, node bounce, OOM kill) can become permanently unbootable with the same data directory.
   
   Found while chaos-testing an integration against Iggy (broker restart mid-stream with durable consumer offsets).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Server 0.9.0 intermittently panics on restart boot replay: 'client_id 0 is reserved for internal use' [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #4361:
URL: https://github.com/apache/iggy/issues/4361#issuecomment-5917905748

   @ZhiXiao-Lin could you please retest using code from https://github.com/apache/iggy/pull/4364 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Server 0.9.0 intermittently panics on restart boot replay: 'client_id 0 is reserved for internal use' [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #4361:
URL: https://github.com/apache/iggy/issues/4361#issuecomment-5912959805

   thanks. i saw this crash couple times in last weeks but i wasn't able to reproduce reliably it locally. i'll check and get back to you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Server 0.9.0 intermittently panics on restart boot replay: 'client_id 0 is reserved for internal use' [iggy]

Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz closed issue #4361: Server 0.9.0 intermittently panics on restart boot replay: 'client_id 0 is reserved for internal use'
URL: https://github.com/apache/iggy/issues/4361


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org