Kafka question on Stream Processing

[Gagan Sabharwal <ga...@gmail.com>]

Hi team,

Say we have a client which has pushed a message to a topic. The message has
a a simple structure

Task - Time of task
Send an email - 1530

Now say that this message is consumed by a consumer subscribed to this
topic.
Since Topic already has a storage, what I intend to do is just read the
message (not poll it) and see if it is before 1530 then send it to the tail
of the partition of that topic. Does Kafka provide such an Api? Next time
when the consumer reads the message and see if the current time is after
1530, it will poll the message and execute the task.

Regards
Gagan

Re: Kafka question on Stream Processing

[Bruno Cadonna <br...@confluent.io>]

Hi Gagan,

If you want to read a message, you need to poll the message from the
broker. The brokers have only very limited notion of message content. They
only know that a message has a key, a value, and some metadata, but they
are not able to interpret the contents of those message components. The
clients are responsible to read and process the messages. For reading and
processing, the clients need to poll the messages from the brokers.

For the processing you want to do, you could use Kafka Streams. See
https://kafka.apache.org/documentation/streams/ for more information. Have
a look at the branch DSL operation there.

Best regards,
Bruno

On Wed, Apr 24, 2019 at 1:54 AM Gagan Sabharwal <ga...@gmail.com> wrote:

> Hi team,
>
> Say we have a client which has pushed a message to a topic. The message has
> a a simple structure
>
> Task - Time of task
> Send an email - 1530
>
> Now say that this message is consumed by a consumer subscribed to this
> topic.
> Since Topic already has a storage, what I intend to do is just read the
> message (not poll it) and see if it is before 1530 then send it to the tail
> of the partition of that topic. Does Kafka provide such an Api? Next time
> when the consumer reads the message and see if the current time is after
> 1530, it will poll the message and execute the task.
>
> Regards
> Gagan
>