You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "mlevkov (via GitHub)" <gi...@apache.org> on 2026/03/12 20:26:28 UTC
[I] bug(connectors): consume() return value discarded in sink runtime [iggy]
mlevkov opened a new issue, #2927:
URL: https://github.com/apache/iggy/issues/2927
## Description
The connector runtime calls `consume()` on each sink plugin via FFI but discards the return value. If a sink's `consume()` returns an error (e.g., HTTP timeout, database connection failure, serialization error), the runtime ignores it — no logging, no retry, no metric. It proceeds to the next poll cycle as if delivery succeeded.
## Location
`core/connectors/runtime/src/sink.rs`, lines 585-593:
```rust
(consume)(
plugin_id,
topic_meta.as_ptr(),
topic_meta.len(),
messages_meta.as_ptr(),
messages_meta.len(),
messages.as_ptr(),
messages.len(),
);
```
The FFI function returns an `i32` (0 = success, 1 = error), but the return value is never checked.
## Impact
This affects **all 7 sink connectors** (Elasticsearch, Iceberg, MongoDB, PostgreSQL, Quickwit, Stdout, and the new HTTP sink). Every `Err(...)` returned by any sink's `consume()` implementation is silently swallowed by the runtime.
As a workaround, sink implementations must log errors internally before returning `Err`. The HTTP sink (#2925) documents this limitation and logs all errors at `error!` level inside `consume()`.
## Suggested Fix
Check the return value and log at minimum:
```rust
let result = (consume)(
plugin_id,
topic_meta.as_ptr(),
topic_meta.len(),
messages_meta.as_ptr(),
messages_meta.len(),
messages.as_ptr(),
messages.len(),
);
if result != 0 {
error!(
"Sink plugin {} consume() returned error code {}",
plugin_id, result
);
// Optionally: increment error metric, trigger retry, etc.
}
```
## Related
- Discussed in #2901 (HTTP sink connector proposal)
- Related to #2926 (offset commit ordering)
- HTTP sink PR: #2925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "drifting-shard (via GitHub)" <gi...@apache.org>.
drifting-shard commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4592650007
Hi maintainers, I’d like to take this issue.
My planned behavior is:
- `0`: treat as success and count the prepared batch as processed
- `1`: treat as sink `consume()` failur; log information, increment sink error metrics, processed_msgs+=0, and keep the consume loop alive
- `-1`: treat as ABI failure, log context, increment error metrics, and return an error so the connector is marked unhealthy
- any unexpected status: handle as an ABI contract violation rather than silently dropping the batch
If that sounds useful, I can open the discussion and summarize the prior PR feedback there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4595646761
Could I please be assigned this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4702550907
Today the sink runtime configures consumers with AutoCommit::When(AutoCommitWhen::PollingMessages) ([runtime/src/sink.rs:421](https://github.com/apache/iggy/discussions/runtime/src/sink.rs#L421)), meaning offsets are committed when messages are polled — before they are processed. A failed batch is silently lost.
**Sink must have an option to decide exactly when it is safe to advance the offset, as only the sink knows when a write is truly durable from the target's perspective.**
This decision of what to do with the i32 returned by the connector (act vs keep it cosmetic) affects enabling the sink to drive offset commits as well.
cc @spetz @hubcio
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "justinmclean (via GitHub)" <gi...@apache.org>.
justinmclean commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-6076362221
All yours to work on, @mlevkov. If you have any questions, just ask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4592415229
after all these attempts, I think this issue needs to be opened as discussion where we dump ideas how to fix this correctly. if anyone feels like doing so, please do it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4699245941
Been thinking about this and adjacent issues of DLQ and offset auto-commit (#2928). This actually requires a decision around responsibility -- **Essentially, a sink will fail and require some action. Who is going to be responsible for the action?**
For ex. who is responsible for retries - sink or runtime? who is responsible for routing to DLQ - sink or runtime?
Looked at it from this perspective, the sink should be responsible for what to do with messages on failure because sink has most context on the reason for the failure. Sinks has context on semantics it wants to maintain (at least once, at most once), what errors are retryable and what errors need to be routed to the DLQ.
The sink can expose this information to the runtime, for ex. exposing a failure and a retryable flag. The downside is the FFI signature changing. Having the runtime supporting routing to DLQ would also require making the FFI boundary richer to expose messages that should go to the DLQ.
Finally, uniformity becomes a concern when multiple sink implement the same behaviours. This can be achieved using utilities in the SDK such as the retry utility.
Therefore, exposing sink errors to the runtime should remain "cosmetic" for the purpose of observability only.
cc @hubcio
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "avirajkhare00 (via GitHub)" <gi...@apache.org>.
avirajkhare00 commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4064467774
I can take a first pass on this.
I checked the sink runtime path first. In `core/connectors/runtime/src/sink.rs`, `process_messages` serializes the batch and invokes the sink consume callback, but the runtime currently does not appear to use a return value from that callback before counting the batch as processed. That suggests the issue is in the runtime or plugin contract, not only in per-sink logic.
My plan is:
1. Trace the sink callback ABI and confirm what success or failure signal the runtime is supposed to receive from plugins.
2. Update the runtime so failed consume calls are not treated as successfully processed.
3. Add or extend an integration test around the connectors runtime path so the regression is covered.
This one looks a bit more invasive than the script or chart issues, but I am happy to dig into it if maintainers are okay with that direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-6057647095
AFAIR Kriti told us she no longer plans to contribute to iggy in foreseeable future, so i think we should unassign @justin-laserdata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4592769128
hey @drifting-shard
it's not that simple. please read https://github.com/apache/iggy/pull/3180 - i'm afraid we would need a proper DLQ. what you proposed helps only for visibility / metrics. it doesn't resolve the actual issue.
@spetz do u have any comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "justin-laserdata (via GitHub)" <gi...@apache.org>.
justin-laserdata commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-6057632205
@kriti-sc I saw you closed #3954 in September. Do you still want this one, or should I unassign it so someone else can pick it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "avirajkhare00 (via GitHub)" <gi...@apache.org>.
avirajkhare00 commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4153297297
@hubcio can assign it to you so that not multiple people are working towards it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "mlevkov (via GitHub)" <gi...@apache.org>.
mlevkov commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-6070949035
@hubcio you can assign it to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "tmchow (via GitHub)" <gi...@apache.org>.
tmchow commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4153284604
Apologies, I missed @avirajkhare00's earlier comment. Go ahead with this one -- I'll focus on #2928 instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "Standing-Man (via GitHub)" <gi...@apache.org>.
Standing-Man commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4300854360
Hi @hubcio, I’m interested in working on this issue. Could you please assign it to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4595732727
@kriti-sc assigned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "tmchow (via GitHub)" <gi...@apache.org>.
tmchow commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4152986127
I'd like to take this. The fix looks straightforward - `process_messages()` at `sink.rs:585` discards the `i32` return from `consume()`. I'll propagate the error and log non-zero returns. Will open a PR once this is confirmed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "avirajkhare00 (via GitHub)" <gi...@apache.org>.
avirajkhare00 commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4153293823
Hey @tmchow
Please go ahead. I am working on helm chart related issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "tmchow (via GitHub)" <gi...@apache.org>.
tmchow commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-4158331687
Implemented in https://github.com/apache/iggy/pull/3061 - captures the i32 return from consume(), logs non-zero results, and propagates the error via RuntimeError::SinkConsumeFailed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): consume() return value discarded in sink runtime [iggy]
Posted by "justin-laserdata (via GitHub)" <gi...@apache.org>.
justin-laserdata commented on issue #2927:
URL: https://github.com/apache/iggy/issues/2927#issuecomment-6076348986
All yours to work on, @mlevkov. If you have any questions, just ask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org