You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Krzysztof Jankiewicz <ja...@gmail.com> on 2021/06/09 11:10:43 UTC

Kafka and micro-batch processing dilemma

Hi,

I have recently made some experiments with stream data processing systems:
Kafka Streams, Spark Structured Streaming, and Flink.

Spark Structured Streaming has micro-batch architecture.
Kafka Streams is described as having a full stream architecture.

I have noticed a few days ago, that by default Kafka Streams does not
process events in the data stream immediately.
The results are sent to the output approximately every 30 seconds.

So where is the difference between micro-batch and full stream processing
performed by Kafka Streams?

I wonder if I set the Spark Structured Streaming trigger to 30 seconds and
outputMode to update, can I notice any difference in stream data
processing?
Does anyone have any example of a data stream that would show the
differences?

Kind regards
Krzysztof

Re: Kafka and micro-batch processing dilemma

Posted by Steve Howard <st...@confluent.io.INVALID>.

Hi Krzysztof,

You can disable caching...

https://stackoverflow.com/a/50646646

Thanks,

Steve

[image:
https://lh4.googleusercontent.com/yEgMoHngOODopEu7R49J8cPKsXnQLeHkqqsErsUgI9D2Rqfz3S0wtoxfYAJUxgaVGcxuPQs2lEwigBc4s6hcXyAo14uopshTq_eh4KBiZvi-Nb2yWSFTKT4iFCQ8-cLOOw_Rm5R8]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.confluent.io&d=DwMFaQ&c=A-GX6P9ovB1qTBp7iQve2Q&r=Utd0DOMevK4PhTyYQyXeCfximjd_X7dLCRZ4fCszhKE&m=KhvsB9pYz4VOEVRjuZQ3wAnqMS0DBvLknGtl87MPHqw&s=jJxjUduPXUSJdg-LRvxdNTjalgtutz6vQV_Jvzgtt3k&e=>

*Steve Howard*
Senior Solutions Engineer
+1 (614) 867-4386

*Follow us:*  Blog
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.confluent.io_blog&d=DwMFaQ&c=A-GX6P9ovB1qTBp7iQve2Q&r=Utd0DOMevK4PhTyYQyXeCfximjd_X7dLCRZ4fCszhKE&m=KhvsB9pYz4VOEVRjuZQ3wAnqMS0DBvLknGtl87MPHqw&s=aIIVdEfz4ptmdJZJ_2WamhSJr15gfTyGtnFp1z4WSNI&e=>
 • Slack
<https://urldefense.proofpoint.com/v2/url?u=https-3A__slackpass.io_confluentcommunity&d=DwMFaQ&c=A-GX6P9ovB1qTBp7iQve2Q&r=Utd0DOMevK4PhTyYQyXeCfximjd_X7dLCRZ4fCszhKE&m=KhvsB9pYz4VOEVRjuZQ3wAnqMS0DBvLknGtl87MPHqw&s=EwsqBtpwvmokBeh5_vbIG1Ku0ZVQaC0qlKx5Ai8EIdI&e=>
 • Twitter
<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_ConfluentInc&d=DwMFaQ&c=A-GX6P9ovB1qTBp7iQve2Q&r=Utd0DOMevK4PhTyYQyXeCfximjd_X7dLCRZ4fCszhKE&m=KhvsB9pYz4VOEVRjuZQ3wAnqMS0DBvLknGtl87MPHqw&s=c8t8vNoKkf10lbbfNrfgfjjOZsH5E6WKZO2mxmpQzgc&e=>

[image:
https://lh5.googleusercontent.com/tS_vAXkPxuaq0ciXARHHCd5gqn8sy1e1DCdLOW8-Bxebkda5hxtLA_89SE2ETfv1bzq4bEwrPhwCvWabusOV1RCr3PYKesyjLX7R_yRdJEeplXeQ66fPPakhYy95As90_FZ4yo4n]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__developer.confluent.io_&d=DwMFaQ&c=A-GX6P9ovB1qTBp7iQve2Q&r=Utd0DOMevK4PhTyYQyXeCfximjd_X7dLCRZ4fCszhKE&m=KhvsB9pYz4VOEVRjuZQ3wAnqMS0DBvLknGtl87MPHqw&s=PlMAsdiAk7xR4EWTNwog67dbhqtWzn1x9EbWvf3xlaQ&e=>
Focus on building apps and not managing clusters with a scalable, resilient
and secure event streaming platform. *Confluent Cloud* is a *fully-managed*
, *serverless* event streaming platform on *GCP*, *AWS* and *Azure*
with a *99.95%
uptime SLA*. Get started here and receive up to *$50 USD* off your bill
each calendar month for the first three months.
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.confluent.io_confluent-2Dcloud_compare_&d=DwMFaQ&c=A-GX6P9ovB1qTBp7iQve2Q&r=Utd0DOMevK4PhTyYQyXeCfximjd_X7dLCRZ4fCszhKE&m=KhvsB9pYz4VOEVRjuZQ3wAnqMS0DBvLknGtl87MPHqw&s=OLhG7d_dmlDinbrCMaHovphlLxF6jEbMKiqno9bVfQc&e=>

On Wed, Jun 9, 2021 at 12:06 PM Krzysztof Jankiewicz <
jankiewicz.krzysztof@gmail.com> wrote:

> Hi,
>
> I have recently made some experiments with stream data processing systems:
> Kafka Streams, Spark Structured Streaming, and Flink.
>
> Spark Structured Streaming has micro-batch architecture.
> Kafka Streams is described as having a full stream architecture.
>
> I have noticed a few days ago, that by default Kafka Streams does not
> process events in the data stream immediately.
> The results are sent to the output approximately every 30 seconds.
>
> So where is the difference between micro-batch and full stream processing
> performed by Kafka Streams?
>
> I wonder if I set the Spark Structured Streaming trigger to 30 seconds and
> outputMode to update, can I notice any difference in stream data
> processing?
> Does anyone have any example of a data stream that would show the
> differences?
>
> Kind regards
> Krzysztof
>