You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iggy.apache.org by "thyseus (via GitHub)" <gi...@apache.org> on 2025/03/14 10:04:25 UTC
"hook" for incoming message
GitHub user thyseus created a discussion: "hook" for incoming message
I want to discuss with you if it is a good idea to implement a "hook" feature on the iggy server side.
What do i mean with "hook" ?
In case a message is incoming, i want to be able to configure one or several "hooks" that will point to a script file or binary that gets executed by the iggy server.
Why is this useful ?
a) i could do mundane tasks that dont require a real consumer implementation, like logs or just pushing the incoming messages somewhere else, really quickly
b) In my use case i have several applications that are written in python, php, rust and many more. Even some, where there is no SDK for iggy yet (php - i have done a primitive one for myself).
Instead of implementing a consumer for each of this applications i could simply register a "hook" in a stream that executes a `php artisan receive-message '{"payload": "example"}'`. This also has the advantage that my consumers dont need to "poll" the iggy server but gets the message immediately. (I know that many advantages of a proper "consumer" implementation are missing then; but there may be cases where that is ok).
What do you think ?
GitHub link: https://github.com/apache/iggy/discussions/1641
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "thyseus (via GitHub)" <gi...@apache.org>.
GitHub user thyseus added a comment to the discussion: "hook" for incoming message
Actually, my rust skills are not yet there; in the web-ui i would image it like:
and in the cli like
```
Commands:
stream stream operations [aliases: s]
topic topic operations [aliases: t]
partition partition operations [aliases: p]
hooks hook operations [aliases: h]
```
```
hook operations
Usage: iggy hook <COMMAND>
Commands:
create Create a hook with given name
delete Delete hook with given id
get Get details for given hook
list List all hooks
help Print this message or the help of the given subcommand(s)
```
```
Create a hook topic with given name on given topic
Topic ID can be specified as a name or ID
Examples
iggy hook create my-first-hook my-topic "cd /path/to/my/other/project; php artisan execute:command {payload}"
Usage: iggy topic hook [OPTIONS] <HOOK_NAME> <TOPIC_NAME> <COMMAND_TO_EXECUTE>
```
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12529562
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "thyseus (via GitHub)" <gi...@apache.org>.
GitHub user thyseus closed a discussion: "hook" for incoming message
I want to discuss with you if it is a good idea to implement a "hook" feature on the iggy server side.
What do i mean with "hook" ?
In case a message is incoming, i want to be able to configure one or several "hooks" that will point to a script file or binary that gets executed by the iggy server.
Why is this useful ?
a) i could do mundane tasks that dont require a real consumer implementation, like logs or just pushing the incoming messages somewhere else, really quickly
b) In my use case i have several applications that are written in python, php, rust and many more. Even some, where there is no SDK for iggy yet (php - i have done a primitive one for myself).
Instead of implementing a consumer for each of this applications i could simply register a "hook" in a stream that executes a `php artisan receive-message '{"payload": "example"}'`. This also has the advantage that my consumers dont need to "poll" the iggy server but gets the message immediately. (I know that many advantages of a proper "consumer" implementation are missing then; but there may be cases where that is ok).
What do you think ?
GitHub link: https://github.com/apache/iggy/discussions/1641
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa added a comment to the discussion: "hook" for incoming message
+1
We've been thinking about the tiered storage feature, and exploring OpenDAL integration. With that, we should be able to point the incoming messages to external destination (object storage, database, file storage, o11y engines, OLAP DBs etc).
Cc: @spetz @hubcio @numinnex
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12499516
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz added a comment to the discussion: "hook" for incoming message
I've just opened another discussion #1670 as the more generic continuation of this topic.
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12680286
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa added a comment to the discussion: "hook" for incoming message
I'd recommend taking a look at Apache Camel's Connector design. We can allow people to create "connectors" and configure them with the settings. Every Connector can have a config file or we can have single "Connectors" yaml file which will be loaded into a map when Iggy server is started.
So we need to come up with a mechanism of calling these "Connectors" at important places like at the entry point, at the time of writing/persisting or reading etc.
Such Connectors can be configured at Topic level or even at the message/batch level. topic level makes the most sense but there could be use cases where we want to let the Clients decide too
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12534313
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "thyseus (via GitHub)" <gi...@apache.org>.
GitHub user thyseus added a comment to the discussion: "hook" for incoming message
Actually my rust skills *had been* barely enough to get a mini-consumer running that is just passing over the payload like this:
```
let output = Command::new("php")
.current_dir("/home/other-project/dev/shopauskunft")
.args([
"artisan",
"iggy:process",
payload
])
.output()
.expect("failed to execute process");
```
so i am totally fine; with a hook one would not need any consumer at all to achieve something similar.
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12529580
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "ilikepi63 (via GitHub)" <gi...@apache.org>.
GitHub user ilikepi63 added a comment to the discussion: "hook" for incoming message
# Introduction
I'm going to start out by separating by usecase, of which I see a few:
- "Trigger" similar to what you'd find in most SQL databases (for example Postgres' [triggers](https://www.postgresql.org/docs/8.1/triggers.html) )
- A User-defined function (UDF) or User Defined (UDAF) similar to what you'd find in query engines such as Datafusion.
- An embedded program - this is where we'd have something that is embeddable into Iggy and invoked as part of it's functionality.
# Usecases
## Trigger
The primary use case would be @thyseus' initial idea. Some questions to answer:
- Could we potentially attach a call out to a separate user defined function based on arbitrary events inside of Iggy?
- What kinds of permissions/capabilities would these triggers have? ie run executables, make network calls (if so, at what OSI layer?)
- What events would we support?
- What will be the calling convention of some of these triggers?
- How will errors in these triggers be handled?
- Will it be important for these triggers to be sandboxed?
## UDF/UDAF implementation
I've always wanted something like this in a streaming service - the use case primarily being compacted topics. For compacted topics in Kafka and Pulsar, only the most recent value for the specified key is retained and the rest are effectively deleted. If you could embed a UDAF function similar to Datafusion's [implementation](https://github.com/apache/datafusion/blob/d20b6d18b901708e48f965385af2119fed01a4c7/datafusion-examples/examples/advanced_udaf.rs#L132), one could potentially have "derivative" topics. This would also likely reduce the need for a lot of stream processing engines which are external and require quite a bit of development.
## Embedded Program
This usecase surrounds itself with possibly changing core behaviour inside of Iggy that is difficult to configure with a configuration file format such as yaml. For instance, a developer can implement a script or something that will adjust the way segments get written or possibly set up custom logic for writing partitions to a separate durable store.
# Embedded Scripting considerations
All of these usecases can be solved with an implementation of some sort of embedded programming language based on events. I will detail some of the idea I have:
## Webassembly
Webassembly is really nice for sandboxing specifically. However, this doesn't seem to be a really important feature for Iggy as users will likely trust their own code. Webassembly is also not a target for many higher level languages without shipping the interpreter/runtime with it. Wasm also has a reputation for moving _really slow_ with it's specs. As far as I know, this is what ScyllaDB used for [their UDF implementation](https://opensource.docs.scylladb.com/stable/cql/wasm.html)
## Shared Library
This is probably the most straightforward one and allows the most flexibility when it comes to the API. However, there are only a handful of languages that can be compiled to a smallish shared library, so effectively we are limited to some languages that allow that type of interfacing (Rust, C, C++ and Swift come to mind). This is apparently the way Arroyo went with [their implementation of UDF's ](https://www.arroyo.dev/blog/rust-plugin-systems)
## Specific language runtime
We could possibly look at embedding a specific language runtime such as RustPython, rusty-v8 or a lua engine. I don't really favour this approach as usually their is a cost to get the data into the VM and get it executed. This also brings a level of overhead as we need to consider things like language version support and all of that.
# Events
We also need to consider which events might be interesting for developers to be able to "hook" into. This could be:
- Compacting a topic
- Consuming a message.
- Producing a message.
- Deleting data based on retention period.
- Dropping the to the next tier level of storage.
I think that [eBPF](https://ebpf.io/) might give us good insights as they have a really nice framework for
1. Hooking into different events in Linux and
2. compiling a library for use within the kernel.
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12555509
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz added a comment to the discussion: "hook" for incoming message
Besides the tiered-storage, we have the plugin/integration feature planned for the future. We haven't decided yet how the interface would look like, and how the "connectors" could be implemented (rust, wasm, or maybe via some network protocol like grpc).
This is certainly on our roadmap, but if I were to estimate the date, probably no sooner than in a few months.
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12499740
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa added a comment to the discussion: "hook" for incoming message
@thyseus it would be awesome if you could propose a design for implementing this feature. Do you want to put together a proposal?
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12503599
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: "hook" for incoming message
Posted by "thyseus (via GitHub)" <gi...@apache.org>.
GitHub user thyseus edited a comment on the discussion: "hook" for incoming message
Actually my rust skills *had been* barely enough to get a mini-consumer running that is just passing over the payload like this:
```
let output = Command::new("php")
.current_dir("/home/me/other-project/")
.args([
"artisan",
"iggy:process",
payload
])
.output()
.expect("failed to execute process");
```
so i am totally fine; with a hook one would not need any consumer at all to achieve something similar.
GitHub link: https://github.com/apache/iggy/discussions/1641#discussioncomment-12529580
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org