You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Andy <an...@santacruzanalytics.com> on 2022/05/31 04:37:39 UTC

Newbie how to get key/value pojo out of a stream?

All the Kstream examples I have found demonstrate how to use map, filter,
and join on streams. The last step they typically user to() to
publish/produce the results to a new stream

How can I get the data out of the stream? For example I need to send the
data to a legacy data that can not use kafka. Or maybe I want to plot the
data,…

I looked at the java doc and did not find anything

Any idea what I should “google” to to find a code example?

Kind regards

Andy

Re: Newbie how to get key/value pojo out of a stream?

Posted by "Matthias J. Sax" <mj...@apache.org>.

`enable.auto.commit` is a Consumer config and does not apply to Kafka 
Stream.

In Kafka Streams, you basically always have auto commit enabled, and you 
can control how frequently commits happen via `commit.interval.ms`.

Also on `close()` Kafka Streams would commit offsets.


-Matthias

On 5/31/22 12:29 PM, Luca wrote:
> Hi Andy,
> 
> The defaults are sensible enough that, under normal operational conditions, your app should pick up from where it left. To dig a little more into this, I suggest you look into `auto.offset.reset` and `enable.auto.commit` options.
> 
> In case, you do need to reprocess everything, kafka streams comes with a handy reset tool. You can read about it here: https://kafka.apache.org/32/documentation/streams/developer-guide/app-reset-tool.html
> 
> Luca
> 
> On Tue, May 31, 2022, at 5:17 PM, andrew davidson wrote:
>> Thanks Luca
>>
>> This is exactly what I was looking for.
>>
>> On a related note let's say I stop and restart my application. What would I have to do so that the I do not re process events?
>>
>> I am still working through the kstreams 101 tutorial. I have not gotten to the DSL tutorials yet
>>
>> Andy
>>
>> On 5/30/22, 11:16 PM, "Luca" <ci...@lucapette.me> wrote:
>>
>>      Hi Andy,
>>
>>      If I understand your problem correctly, you want a "foreach" terminal operation. You can check out the API here: https://kafka.apache.org/32/documentation/streams/developer-guide/dsl-api.html
>>
>>      Luca
>>
>>      On Tue, May 31, 2022, at 6:37 AM, Andy wrote:
>>      > All the Kstream examples I have found demonstrate how to use map, filter,
>>      > and join on streams. The last step they typically user to() to
>>      > publish/produce the results to a new stream
>>      >
>>      > How can I get the data out of the stream? For example I need to send the
>>      > data to a legacy data that can not use kafka. Or maybe I want to plot the
>>      > data,…
>>      >
>>      > I looked at the java doc and did not find anything
>>      >
>>      > Any idea what I should “google” to to find a code example?
>>      >
>>      > Kind regards
>>      >
>>      > Andy
>>      >
>>
>>      lucapette.me
>>
> 
> lucapette.me
>

Re: Newbie how to get key/value pojo out of a stream?

Posted by Luca <ci...@lucapette.me>.

Hi Andy,

The defaults are sensible enough that, under normal operational conditions, your app should pick up from where it left. To dig a little more into this, I suggest you look into `auto.offset.reset` and `enable.auto.commit` options.

In case, you do need to reprocess everything, kafka streams comes with a handy reset tool. You can read about it here: https://kafka.apache.org/32/documentation/streams/developer-guide/app-reset-tool.html

Luca

On Tue, May 31, 2022, at 5:17 PM, andrew davidson wrote:
> Thanks Luca
> 
> This is exactly what I was looking for.
> 
> On a related note let's say I stop and restart my application. What would I have to do so that the I do not re process events?
> 
> I am still working through the kstreams 101 tutorial. I have not gotten to the DSL tutorials yet
> 
> Andy
> 
> On 5/30/22, 11:16 PM, "Luca" <ci...@lucapette.me> wrote:
> 
>     Hi Andy,
> 
>     If I understand your problem correctly, you want a "foreach" terminal operation. You can check out the API here: https://kafka.apache.org/32/documentation/streams/developer-guide/dsl-api.html
> 
>     Luca
> 
>     On Tue, May 31, 2022, at 6:37 AM, Andy wrote:
>     > All the Kstream examples I have found demonstrate how to use map, filter,
>     > and join on streams. The last step they typically user to() to
>     > publish/produce the results to a new stream
>     > 
>     > How can I get the data out of the stream? For example I need to send the
>     > data to a legacy data that can not use kafka. Or maybe I want to plot the
>     > data,…
>     > 
>     > I looked at the java doc and did not find anything
>     > 
>     > Any idea what I should “google” to to find a code example?
>     > 
>     > Kind regards
>     > 
>     > Andy
>     > 
> 
>     lucapette.me
> 

lucapette.me

Re: Newbie how to get key/value pojo out of a stream?

Posted by andrew davidson <an...@santacruzanalytics.com>.

Thanks Luca

This is exactly what I was looking for.

On a related note let's say I stop and restart my application. What would I have to do so that the I do not re process events?

I am still working through the kstreams 101 tutorial. I have not gotten to the DSL tutorials yet

Andy

On 5/30/22, 11:16 PM, "Luca" <ci...@lucapette.me> wrote:

    Hi Andy,

    If I understand your problem correctly, you want a "foreach" terminal operation. You can check out the API here: https://kafka.apache.org/32/documentation/streams/developer-guide/dsl-api.html

    Luca

    On Tue, May 31, 2022, at 6:37 AM, Andy wrote:
    > All the Kstream examples I have found demonstrate how to use map, filter,
    > and join on streams. The last step they typically user to() to
    > publish/produce the results to a new stream
    > 
    > How can I get the data out of the stream? For example I need to send the
    > data to a legacy data that can not use kafka. Or maybe I want to plot the
    > data,…
    > 
    > I looked at the java doc and did not find anything
    > 
    > Any idea what I should “google” to to find a code example?
    > 
    > Kind regards
    > 
    > Andy
    > 

    lucapette.me

Re: Newbie how to get key/value pojo out of a stream?

Posted by Luca <ci...@lucapette.me>.

Hi Andy,

If I understand your problem correctly, you want a "foreach" terminal operation. You can check out the API here: https://kafka.apache.org/32/documentation/streams/developer-guide/dsl-api.html

Luca

On Tue, May 31, 2022, at 6:37 AM, Andy wrote:
> All the Kstream examples I have found demonstrate how to use map, filter,
> and join on streams. The last step they typically user to() to
> publish/produce the results to a new stream
> 
> How can I get the data out of the stream? For example I need to send the
> data to a legacy data that can not use kafka. Or maybe I want to plot the
> data,…
> 
> I looked at the java doc and did not find anything
> 
> Any idea what I should “google” to to find a code example?
> 
> Kind regards
> 
> Andy
> 

lucapette.me