Multi topic poling / wildcard topic polling (hierachy)

["hubcio (via GitHub)" <gi...@apache.org>]

GitHub user hubcio created a discussion: Multi topic poling / wildcard topic polling (hierachy)

the idea: poll many topics at once with a pattern instead of one per call. something like `*.*.*.*.<domain>.<tenant>`, matching every topic that fits, including ones created later. NATS, MQTT and Solace all expose this (as subjects / topic filters) and people lean on it for event-driven systems.

we've had a couple of asks for it now, the latest wanting to swap a NATS event bus for iggy.
this isn't about storage. matching happens at poll time against live metadata, so there is nothing new to persist (and naming topics `a.b.c.d` already works today).

we could fake it client-side: list topics, filter, poll each. but then you have to re-list constantly to catch new topics, and it's a lot of round trips. this feels like something we should own server-side: a poll takes a pattern, the server resolves it against current metadata and returns messages from every matching topic.

the interesting questions:

- offsets: the per-topic/partition cursors already exist, but a multi-topic poll response has to carry many of them at once (today's response is single-topic).
- consumer groups: a newly matched topic has no committed offset for the group, so who gets its partitions, and where do they start reading? (the rebalance / revocation rabbit hole)
- bounding the response when a pattern fans out to hundreds of topics. this probably wants the deferred-poll work (#2854) first.
- syntax: token wildcards like NATS/MQTT (`*`, `>`/`#`) vs full regex. i lean token, since it's predictable and cheap to match.
- scope: do we match within a single stream, or across streams?

no rush on timing. realistically this lands after VSR, and i'd put it roughly a notch below #2854 in priority. mostly i'm opening this to see who wants it, what their topic patterns look like, and to settle the questions above before anyone writes code.

(moved out of #3505, which i'm closing.)

thoughts?

GitHub link: https://github.com/apache/iggy/discussions/3570

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org

Re: Multi topic poling / wildcard topic polling (hierachy)

["spetz (via GitHub)" <gi...@apache.org>]

GitHub user spetz added a comment to the discussion: Multi topic poling / wildcard topic polling (hierachy)

I think that such a feature could be really useful for some scenarios.

Also, my question is whether there should be an aggregated poll response (a new model), or maybe we could extend the existing `PolledMessages` with `stream_id: u32` + `topic_id: u32` - in such a case, whether you poll messages from the single topic or multiple topics, each request to the server could always return `PolledMessages` with the different stream & topic ID pair?

At the same time, we'd either need to change the `PollMessages` or introduce another command, like some kind of `RegisterSubscripion` for the given streams + topics returning some unique ID of the registration, and then we could pass it along in `PollMessages` (via some new field) to keep polling from the multiple sources?

As for the consumer groups, such a consumer should join all of the requested topics and whether to auto ack or not during  `PollMessages` should remain as a single flag applicable to all CG.

GitHub link: https://github.com/apache/iggy/discussions/3570#discussioncomment-17498277

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org