You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jerry Peng <je...@gmail.com> on 2025/05/28 04:13:11 UTC
[DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Hi all,
I want to start a discussion thread for the SPIP titled “Real-Time Mode in
Apache Spark Structured Streaming” that I've been working on with Siying
Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [JIRA
<https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
<https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
].
The SPIP proposes a new execution mode called “Real-time Mode” in Spark
Structured Streaming that significantly lowers end-to-end latency for
processing streams of data.
A key principle of this proposal is compatibility. Our goal is to make
Spark capable of handling streaming jobs that need results almost
immediately (within O(100) milliseconds). We want to achieve this without
changing the high-level DataFrame/Dataset API that users already use – so
existing streaming queries can run in this new ultra-low-latency mode by
simply turning it on, without rewriting their logic.
In short, we’re trying to enable Spark to power real-time applications
(like instant anomaly alerts or live personalization) that today cannot
meet their latency requirements with Spark’s current streaming engine.
We'd greatly appreciate your feedback, thoughts, and suggestions on this
approach!
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by "L. C. Hsieh" <vi...@gmail.com>.
Thanks to everyone in the community for your interest and support for this
proposal. We've had extensive and constructive discussions both in this
thread and in the SPIP document. These conversations have been positive and
encouraging for moving in this direction. Special thanks to the SPIP
authors for actively addressing questions and feedback.
Unless something has been overlooked, it seems that the discussions have
largely settled. The next step will be to initiate a vote on the SPIP. To
allow time for any final comments or feedback, I will hold off on starting
the vote for another day or two.
On Fri, May 30, 2025 at 10:56 AM Denny Lee <de...@gmail.com> wrote:
> +1 (non-binding)
>
> On Fri, May 30, 2025 at 9:17 AM xianjin <xi...@apache.org> wrote:
>
>> +1
>> Sent from my iPhone
>>
>> On May 29, 2025, at 12:53 PM, Yuanjian Li <xy...@gmail.com> wrote:
>>
>>
>> +1
>>
>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>
>>> +1, LGTM.
>>>
>>> Kent
>>>
>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>
>>>> +1. Super excited by this initiative!
>>>>
>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>>>>> eliminate the need for separate streaming engines, reducing system
>>>>>> complexity and operational cost. Excited to see this direction!
>>>>>>
>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> My point about "in real time application or data, there is nothing
>>>>>>> as an answer which is supposed to be late and correct. The timeliness is
>>>>>>> part of the application. if I get the right answer too slowly it becomes
>>>>>>> useless or wrong" is actually fundamental to *why* we need this
>>>>>>> Spark Structured Streaming proposal.
>>>>>>>
>>>>>>> The proposal is precisely about enabling Spark to power applications
>>>>>>> where, as I define it, the *timeliness* of the answer is as
>>>>>>> critical as its *correctness*. Spark's current streaming engine,
>>>>>>> primarily operating on micro-batches, often delivers results that are
>>>>>>> technically "correct" but arrive too late to be truly useful for certain
>>>>>>> high-stakes, real-time scenarios. This makes them "useless or wrong" in a
>>>>>>> practical, business-critical sense.
>>>>>>>
>>>>>>> For example *in real-time fraud detection* and In *high-frequency
>>>>>>> trading,* market data or trade execution commands must be delivered
>>>>>>> with minimal latency. Even a slight delay can mean missed opportunities or
>>>>>>> significant financial losses, making a "correct" price update useless if
>>>>>>> it's not instantaneous. able for these demanding use cases, where a
>>>>>>> "late but correct" answer is simply not good enough. As a colliery it is a
>>>>>>> fundamental concept, so it has to be treated as such not as a
>>>>>>> comment.in SPIP
>>>>>>>
>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>> Dr Mich Talebzadeh,
>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Mich,
>>>>>>>>
>>>>>>>> Sorry, I may be missing something here but what does your
>>>>>>>> definition here have to do with the SPIP? Perhaps add comments directly
>>>>>>>> to the SPIP to provide context as the code snippet below is a direct copy
>>>>>>>> from the SPIP itself.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Denny
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> just to add
>>>>>>>>>
>>>>>>>>> A stronger definition of real time. The engineering definition of
>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>>
>>>>>>>>> However, I put a stronger definition. In real time application or
>>>>>>>>> data, there is nothing as an answer which is supposed to be late and
>>>>>>>>> correct. The timeliness is part of the application.if I get the right
>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>> GDPR
>>>>>>>>>
>>>>>>>>> view my Linkedin profile
>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> The current limitations in SSS come from micro-batching.If you
>>>>>>>>>> are going to reduce micro-batching, this reduction must be balanced against
>>>>>>>>>> the available processing capacity of the cluster to prevent back pressure
>>>>>>>>>> and instability. In the case of Continuous Processing mode, a
>>>>>>>>>> specific continuous trigger with a desired checkpoint interval quote
>>>>>>>>>>
>>>>>>>>>> "
>>>>>>>>>> df.writeStream
>>>>>>>>>> .format("...")
>>>>>>>>>> .option("...")
>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger
>>>>>>>>>> type to enable real-time Mode
>>>>>>>>>> .start()
>>>>>>>>>> This Trigger.RealTime signals that the query should run in the
>>>>>>>>>> new ultra low-latency execution mode. A time interval can also be
>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each micro-batch should
>>>>>>>>>> run for.
>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>>>> HTH
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>>> GDPR
>>>>>>>>>>
>>>>>>>>>> view my Linkedin profile
>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that I've been
>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and
>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>>>>>> ].
>>>>>>>>>>>
>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode”
>>>>>>>>>>> in Spark Structured Streaming that significantly lowers end-to-end latency
>>>>>>>>>>> for processing streams of data.
>>>>>>>>>>>
>>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is
>>>>>>>>>>> to make Spark capable of handling streaming jobs that need results almost
>>>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>>>>
>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>>>>>> engine.
>>>>>>>>>>>
>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions
>>>>>>>>>>> on this approach!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>
>>>>> --
>>>>> Best,
>>>>> Yanbo
>>>>>
>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Denny Lee <de...@gmail.com>.
+1 (non-binding)
On Fri, May 30, 2025 at 9:17 AM xianjin <xi...@apache.org> wrote:
> +1
> Sent from my iPhone
>
> On May 29, 2025, at 12:53 PM, Yuanjian Li <xy...@gmail.com> wrote:
>
>
> +1
>
> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>
>> +1, LGTM.
>>
>> Kent
>>
>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>
>>> +1. Super excited by this initiative!
>>>
>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1
>>>>> By unifying batch and low-latency streaming in Spark, we can eliminate
>>>>> the need for separate streaming engines, reducing system complexity and
>>>>> operational cost. Excited to see this direction!
>>>>>
>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> My point about "in real time application or data, there is nothing as
>>>>>> an answer which is supposed to be late and correct. The timeliness is part
>>>>>> of the application. if I get the right answer too slowly it becomes useless
>>>>>> or wrong" is actually fundamental to *why* we need this Spark
>>>>>> Structured Streaming proposal.
>>>>>>
>>>>>> The proposal is precisely about enabling Spark to power applications
>>>>>> where, as I define it, the *timeliness* of the answer is as critical
>>>>>> as its *correctness*. Spark's current streaming engine, primarily
>>>>>> operating on micro-batches, often delivers results that are technically
>>>>>> "correct" but arrive too late to be truly useful for certain high-stakes,
>>>>>> real-time scenarios. This makes them "useless or wrong" in a practical,
>>>>>> business-critical sense.
>>>>>>
>>>>>> For example *in real-time fraud detection* and In *high-frequency
>>>>>> trading,* market data or trade execution commands must be delivered
>>>>>> with minimal latency. Even a slight delay can mean missed opportunities or
>>>>>> significant financial losses, making a "correct" price update useless if
>>>>>> it's not instantaneous. able for these demanding use cases, where a
>>>>>> "late but correct" answer is simply not good enough. As a colliery it is a
>>>>>> fundamental concept, so it has to be treated as such not as a
>>>>>> comment.in SPIP
>>>>>>
>>>>>> Hope this clarifies the connection in practical terms
>>>>>> Dr Mich Talebzadeh,
>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey Mich,
>>>>>>>
>>>>>>> Sorry, I may be missing something here but what does your definition
>>>>>>> here have to do with the SPIP? Perhaps add comments directly to the SPIP
>>>>>>> to provide context as the code snippet below is a direct copy from the SPIP
>>>>>>> itself.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Denny
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>
>>>>>>>> just to add
>>>>>>>>
>>>>>>>> A stronger definition of real time. The engineering definition of
>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>
>>>>>>>> However, I put a stronger definition. In real time application or
>>>>>>>> data, there is nothing as an answer which is supposed to be late and
>>>>>>>> correct. The timeliness is part of the application.if I get the right
>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>> GDPR
>>>>>>>>
>>>>>>>> view my Linkedin profile
>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> The current limitations in SSS come from micro-batching.If you are
>>>>>>>>> going to reduce micro-batching, this reduction must be balanced against the
>>>>>>>>> available processing capacity of the cluster to prevent back pressure and
>>>>>>>>> instability. In the case of Continuous Processing mode, a
>>>>>>>>> specific continuous trigger with a desired checkpoint interval quote
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> df.writeStream
>>>>>>>>> .format("...")
>>>>>>>>> .option("...")
>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger
>>>>>>>>> type to enable real-time Mode
>>>>>>>>> .start()
>>>>>>>>> This Trigger.RealTime signals that the query should run in the new
>>>>>>>>> ultra low-latency execution mode. A time interval can also be specified,
>>>>>>>>> e.g. “300 Seconds”, to indicate how long each micro-batch should run for.
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>>> HTH
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>> GDPR
>>>>>>>>>
>>>>>>>>> view my Linkedin profile
>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that I've been
>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and
>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>>>>> ].
>>>>>>>>>>
>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
>>>>>>>>>> Spark Structured Streaming that significantly lowers end-to-end latency for
>>>>>>>>>> processing streams of data.
>>>>>>>>>>
>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is to
>>>>>>>>>> make Spark capable of handling streaming jobs that need results almost
>>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>>>
>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>>>>> engine.
>>>>>>>>>>
>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions
>>>>>>>>>> on this approach!
>>>>>>>>>>
>>>>>>>>>>
>>>>
>>>> --
>>>> Best,
>>>> Yanbo
>>>>
>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by xianjin <xi...@apache.org>.
+1
Sent from my iPhone
> On May 29, 2025, at 12:53 PM, Yuanjian Li <xy...@gmail.com> wrote:
>
>
>
>
> +1
>
>
>
>
>
> Kent Yao <[yao@apache.org](mailto:yao@apache.org)> 于2025年5月28日周三 19:31写道：
>
>
>> +1, LGTM.
>>
>>
>
>>
>> Kent
>>
>>
> 在 2025年5月29日星期四，Chao Sun <[sunchao@apache.org](mailto:sunchao@apache.org)>
> 写道：
>
>>
>>> +1. Super excited by this initiative!
>
>>>
>>>
>
>>>
>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang
<[ybliang8@gmail.com](mailto:ybliang8@gmail.com)> wrote:
>
>>>
>>>> +1
>
>>>>
>>>>
>
>>>>
>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao
<[huaxin.gao11@gmail.com](mailto:huaxin.gao11@gmail.com)> wrote:
>
>>>>
>>>>> +1
>>>>>
>>>>> By unifying batch and low-latency streaming in Spark, we can eliminate
the need for separate streaming engines, reducing system complexity and
operational cost. Excited to see this direction!
>>>>>
>>>>>
>
>>>>>
>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh
<[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:
>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> My point about "in real time application or data, there is nothing as
an answer which is supposed to be late and correct. The timeliness is part of
the application. if I get the right answer too slowly it becomes useless or
wrong" is actually fundamental to _why_ we need this Spark Structured
Streaming proposal.
>>>>>>
>>>>>> The proposal is precisely about enabling Spark to power applications
where, as I define it, the _timeliness_ of the answer is as critical as its
_correctness_. Spark's current streaming engine, primarily operating on micro-
batches, often delivers results that are technically "correct" but arrive too
late to be truly useful for certain high-stakes, real-time scenarios. This
makes them "useless or wrong" in a practical, business-critical sense.
>>>>>>
>>>>>> For example _in real-time fraud detection_ and In _high-frequency
trading,_ market data or trade execution commands must be delivered with
minimal latency. Even a slight delay can mean missed opportunities or
significant financial losses, making a "correct" price update useless if it's
not instantaneous. able for these demanding use cases, where a "late but
correct" answer is simply not good enough. As a colliery it is a fundamental
concept, so it has to be treated as such not as a
[comment.in](http://comment.in) SPIP
>>>>>>
>>>>>> Hope this clarifies the connection in practical terms
>>>>>>
>>>>>> Dr Mich Talebzadeh,
>>>>>>
>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> 
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)
>>>>>>
>>>>>>
>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee
<[denny.g.lee@gmail.com](mailto:denny.g.lee@gmail.com)> wrote:
>
>>>>>>
>>>>>>> Hey Mich,
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> Sorry, I may be missing something here but what does your definition
here have to do with the SPIP? Perhaps add comments directly to the SPIP to
provide context as the code snippet below is a direct copy from the SPIP
itself.
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Denny
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh
<[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:
>
>>>>>>>
>>>>>>>> just to add
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> A stronger definition of real time. The engineering definition of
real time is roughly fast enough to be interactive
>>>>>>>>
>>>>>>>> However, I put a stronger definition. In real time application or
data, there is nothing as an answer which is supposed to be late and correct.
The timeliness is part of the application.if I get the right answer too slowly
it becomes useless or wrong
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> 
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh
<[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:
>
>>>>>>>>
>>>>>>>>> The current limitations in SSS come from micro-batching.If you are
going to reduce micro-batching, this reduction must be balanced against the
available processing capacity of the cluster to prevent back pressure and
instability. In the case of Continuous Processing mode, a specific continuous
trigger with a desired checkpoint interval quote
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> df.writeStream
> .format("...")
> .option("...")
> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type to
> enable real-time Mode
> .start()
> This Trigger.RealTime signals that the query should run in the new ultra
> low-latency execution mode. A time interval can also be specified, e.g.
> “300 Seconds”, to indicate how long each micro-batch should run for.
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>>>
>>>>>>>>> HTH
>>>>>>>>>
>>>>>>>>>
>
>
>>>>>>>>>
>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>
>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
GDPR
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> 
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng
<[jerry.boyang.peng@gmail.com](mailto:jerry.boyang.peng@gmail.com)> wrote:
>
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> I want to start a discussion thread for the SPIP titled “Real-Time
Mode in Apache Spark Structured Streaming” that I've been working on with
Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust:
[[JIRA](https://issues.apache.org/jira/browse/SPARK-52330)]
[[Doc](https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing)].
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
Spark Structured Streaming that significantly lowers end-to-end latency for
processing streams of data.
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is to
make Spark capable of handling streaming jobs that need results almost
immediately (within O(100) milliseconds). We want to achieve this without
changing the high-level DataFrame/Dataset API that users already use – so
existing streaming queries can run in this new ultra-low-latency mode by
simply turning it on, without rewriting their logic.
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
applications (like instant anomaly alerts or live personalization) that today
cannot meet their latency requirements with Spark’s current streaming engine.
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions on
this approach!
>>>>>>>>>>
>>>>>>>>>>
>
>>>>
>>>>
>
>>>>
>>>>
>
>>>>
>>>> \--
>
>>>>
>>>> Best,
>>>>
>>>> Yanbo
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jerry Peng <je...@gmail.com>.
Mich,
Sounds good. I will add the clarification to the SPIP.
On Fri, May 30, 2025 at 3:47 AM Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi Jerry,
>
> In essence, these definitions (hard or soft) help clarify that "real-time"
> is* not a single, monolithic concept here,* but rather a spectrum defined
> by the criticality of timeliness and systems under consideration. Common
> data processing solutions branded as "real-time" are typically operating on
> the softer end of this spectrum, providing performance crucial for
> applications under considerations (for example within SLAs) where delays
> are undesirable but not show stopper.
>
> I therefore suggest the SPIP should mention this explicitly, so we can
> move on
>
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Fri, 30 May 2025 at 07:57, Jerry Peng <je...@gmail.com>
> wrote:
>
>> Mark,
>>
>> For real-time systems there is a concept of "soft" real-time and "hard"
>> real-time systems. These concepts exist in textbooks. Here is a document
>> by intel that explains it:
>>
>>
>> https://www.intel.com/content/www/us/en/learn/what-is-a-real-time-system.html
>>
>> "In a soft real-time system, computers or equipment will continue to
>> function after a missed deadline but may produce a lower-quality output.
>> For example, latency in online video games can impact player interactions,
>> but otherwise present no serious consequences."
>>
>> "Hard real-time systems have zero delay tolerance, and delayed signals
>> can result in total failure or present immediate danger to users. Flight
>> control systems and pacemakers are both examples where timeliness is not
>> only essential but the lack of it can result in a life-or-death situation."
>>
>> I don't think it is inaccurate or misleading to call this mode
>> real-time. It is soft real-time.
>>
>> On Thu, May 29, 2025 at 11:44 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> Clarifying what is meant by "real-time" and explicitly differentiating
>>> it from actual real-time computing should be a bare minimum. I still don't
>>> like the use of marketing-speak "real-time" that isn't really real-time in
>>> engineering documents or API namespaces.
>>>
>>> On Thu, May 29, 2025 at 10:43 PM Jerry Peng <je...@gmail.com>
>>> wrote:
>>>
>>>> Mark,
>>>>
>>>> I thought we are simply discussing the naming of the mode? Like I
>>>> mentioned, if you think simply calling this mode "real-time" mode may cause
>>>> confusion because "real-time" can mean other things in other fields, I can
>>>> clarify what we mean by "real-time" explicitly in the SPIP document and any
>>>> future documentation. That is not a problem and thank you for your feedback.
>>>>
>>>> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Referencing other misuse of "real-time" is not persuasive. A SPIP is
>>>>> an engineering document, not a marketing document. Technical clarity and
>>>>> accuracy should be non-negotiable.
>>>>>
>>>>>
>>>>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <
>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>
>>>>>> Mark,
>>>>>>
>>>>>> As an example of my point if you go the the Apache Storm (another
>>>>>> stream processing engine) website:
>>>>>>
>>>>>> https://storm.apache.org/
>>>>>>
>>>>>> It describes Storm as:
>>>>>>
>>>>>> "Apache Storm is a free and open source distributed *realtime*
>>>>>> computation system"
>>>>>>
>>>>>> If you can to apache Flink:
>>>>>>
>>>>>>
>>>>>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>>>>>
>>>>>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>>>>>
>>>>>> Thus, what the term "rea-time" implies in this should not be
>>>>>> confusing for folks in this area.
>>>>>>
>>>>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <
>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>
>>>>>>> Mich,
>>>>>>>
>>>>>>> If I understood your last email correctly, I think you also wanted
>>>>>>> to have a discussion about naming? Why are we calling this new execution
>>>>>>> mode described in the SPIP "Real-time Mode"? Here are my two cents.
>>>>>>> Firstly, "continuous mode" is taken and we want another name to describe an
>>>>>>> execution mode that provides ultra low latency processing. We could have
>>>>>>> called it "low latency mode", though I don't really like that naming since
>>>>>>> it implies the other execution modes are not low latency which I don't
>>>>>>> believe is true. This new proposed mode can simply deliver even lower
>>>>>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>>>>>> are talking about "soft" real-time here. I think when we are talking about
>>>>>>> distributed stream processing systems in the space of big data analytics,
>>>>>>> it is reasonable to assume anything described in this space as "real-time"
>>>>>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>>>>>> can provide clear documentation on what "real-time" in real-time mode means
>>>>>>> and what it guarantees. Just my thoughts. I would love to hear other
>>>>>>> perspectives.
>>>>>>>
>>>>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>
>>>>>>>> I think from what I have seen there are a good number of +1
>>>>>>>> responses as opposed to quantitative discussions (based on my observations
>>>>>>>> only). Given the objectives of the thread, we ought to focus on what is
>>>>>>>> meant by real time compared to continuous modes.To be fair, it is a
>>>>>>>> common point of confusion, and the terms are often used interchangeably in
>>>>>>>> general conversation, but in technical contexts, especially with streaming
>>>>>>>> data platforms, they have specific and important differences.
>>>>>>>>
>>>>>>>> "Continuous Mode" refers to a processing strategy that aims for
>>>>>>>> true, uninterrupted, sub-millisecond latency processing. Chiefly
>>>>>>>>
>>>>>>>> - Event-at-a-Time (or very small batch groups): The system
>>>>>>>> processes individual events or extremely small groups of events ->
>>>>>>>> microbatches as they flow through the pipeline.
>>>>>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>>>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>>>>>> even below
>>>>>>>> - Most business use cases (say financial markets) can live with
>>>>>>>> this as they do not rely on rdges
>>>>>>>>
>>>>>>>> Now what is meant by "Real-time Mode"
>>>>>>>>
>>>>>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>>>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>>>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>>>>>> that achieves real-time characteristics but might do so differently or more
>>>>>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>>>>>> mention, in real time application , there is nothing as an answer which is
>>>>>>>> supposed to be late and correct. The timeliness is part of the application.
>>>>>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>>>>>> call the "Late and Correct is Useless" Principle
>>>>>>>>
>>>>>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>>>>>> delivers low-latency processing with high reliability and ease of use,
>>>>>>>> leveraging established, battle-tested components.I invite the audience to
>>>>>>>> have a discussion on this.
>>>>>>>>
>>>>>>>> HTH
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>> GDPR
>>>>>>>>
>>>>>>>> view my Linkedin profile
>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>>>>>> > +1
>>>>>>>>> >
>>>>>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>>>>>> >
>>>>>>>>> > > +1.
>>>>>>>>> > >
>>>>>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com>
>>>>>>>>> wrote:
>>>>>>>>> > >
>>>>>>>>> > >> +1
>>>>>>>>> > >> Sent from my iPhone
>>>>>>>>> > >>
>>>>>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >> +1 Nice feature
>>>>>>>>> > >>
>>>>>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>>>>>> xyliyuanjian@gmail.com>
>>>>>>>>> > >> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>> +1
>>>>>>>>> > >>>
>>>>>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>>>>>> > >>>
>>>>>>>>> > >>>> +1, LGTM.
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> Kent
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>>>>> > >>>>
>>>>>>>>> > >>>>> +1. Super excited by this initiative!
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>>>>>> ybliang8@gmail.com>
>>>>>>>>> > >>>>> wrote:
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>>> +1
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>>>>>> huaxin.gao11@gmail.com>
>>>>>>>>> > >>>>>> wrote:
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>>> +1
>>>>>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we
>>>>>>>>> can
>>>>>>>>> > >>>>>>> eliminate the need for separate streaming engines,
>>>>>>>>> reducing system
>>>>>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>>>>>> direction!
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>>> Hi,
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> My point about "in real time application or data, there
>>>>>>>>> is nothing
>>>>>>>>> > >>>>>>>> as an answer which is supposed to be late and correct.
>>>>>>>>> The timeliness is
>>>>>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>>>>>> slowly it becomes
>>>>>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we
>>>>>>>>> need this
>>>>>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of
>>>>>>>>> the answer
>>>>>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>>>>>> streaming
>>>>>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>>>>>> delivers results that
>>>>>>>>> > >>>>>>>> are technically "correct" but arrive too late to be
>>>>>>>>> truly useful for
>>>>>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes
>>>>>>>>> them "useless or
>>>>>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>>>>>> *high-frequency
>>>>>>>>> > >>>>>>>> trading,* market data or trade execution commands must
>>>>>>>>> be
>>>>>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>>>>>> mean missed
>>>>>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>>>>>> "correct" price
>>>>>>>>> > >>>>>>>> update useless if it's not instantaneous. able for
>>>>>>>>> these demanding
>>>>>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply
>>>>>>>>> not good enough. As
>>>>>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>>>>>> treated as such not
>>>>>>>>> > >>>>>>>> as a comment.in SPIP
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>>> Analysis |
>>>>>>>>> > >>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>>>>>> denny.g.lee@gmail.com>
>>>>>>>>> > >>>>>>>> wrote:
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>> Hey Mich,
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does
>>>>>>>>> your
>>>>>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps
>>>>>>>>> add comments directly
>>>>>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet
>>>>>>>>> below is a direct copy
>>>>>>>>> > >>>>>>>>> from the SPIP itself.
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>> > >>>>>>>>> Denny
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>> just to add
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>>>>>> definition of
>>>>>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>>>>>> application or
>>>>>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed
>>>>>>>>> to be late and
>>>>>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if
>>>>>>>>> I get the right
>>>>>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>>> Analysis |
>>>>>>>>> > >>>>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>>>>>> micro-batching.If you
>>>>>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction
>>>>>>>>> must be balanced against
>>>>>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>>>>>> prevent back pressure
>>>>>>>>> > >>>>>>>>>>> and instability. In the case of Continuous
>>>>>>>>> Processing mode, a
>>>>>>>>> > >>>>>>>>>>> specific continuous trigger with a desired
>>>>>>>>> checkpoint interval quote
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> "
>>>>>>>>> > >>>>>>>>>>> df.writeStream
>>>>>>>>> > >>>>>>>>>>> .format("...")
>>>>>>>>> > >>>>>>>>>>> .option("...")
>>>>>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) //
>>>>>>>>> new trigger
>>>>>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>>>>>> > >>>>>>>>>>> .start()
>>>>>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should
>>>>>>>>> run in the
>>>>>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time
>>>>>>>>> interval can also be
>>>>>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long
>>>>>>>>> each micro-batch should
>>>>>>>>> > >>>>>>>>>>> run for.
>>>>>>>>> > >>>>>>>>>>> "
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that
>>>>>>>>> simple
>>>>>>>>> > >>>>>>>>>>> HTH
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime |
>>>>>>>>> Forensic Analysis |
>>>>>>>>> > >>>>>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> Hi all,
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP
>>>>>>>>> titled
>>>>>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured
>>>>>>>>> Streaming” that I've been
>>>>>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao
>>>>>>>>> Sun, Jungtaek Lim, and
>>>>>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>>>>>> [Doc
>>>>>>>>> > >>>>>>>>>>>> <
>>>>>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>>>>>> >
>>>>>>>>> > >>>>>>>>>>>> ].
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>>>>>> “Real-time Mode”
>>>>>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly
>>>>>>>>> lowers end-to-end latency
>>>>>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility.
>>>>>>>>> Our goal is
>>>>>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs
>>>>>>>>> that need results almost
>>>>>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want
>>>>>>>>> to achieve this without
>>>>>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>>>>>> users already use – so
>>>>>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>>>>>> ultra-low-latency mode by
>>>>>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>>>>>> real-time
>>>>>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>>>>>> personalization) that
>>>>>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>>>>>> Spark’s current streaming
>>>>>>>>> > >>>>>>>>>>>> engine.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> --
>>>>>>>>> > >>>>>> Best,
>>>>>>>>> > >>>>>> Yanbo
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>
>>>>>>>>> > >>
>>>>>>>>> > >> --
>>>>>>>>> > >> John Zhuge
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>
>>>>>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Jerry,
In essence, these definitions (hard or soft) help clarify that "real-time"
is* not a single, monolithic concept here,* but rather a spectrum defined
by the criticality of timeliness and systems under consideration. Common
data processing solutions branded as "real-time" are typically operating on
the softer end of this spectrum, providing performance crucial for
applications under considerations (for example within SLAs) where delays
are undesirable but not show stopper.
I therefore suggest the SPIP should mention this explicitly, so we can
move on
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Fri, 30 May 2025 at 07:57, Jerry Peng <je...@gmail.com>
wrote:
> Mark,
>
> For real-time systems there is a concept of "soft" real-time and "hard"
> real-time systems. These concepts exist in textbooks. Here is a document
> by intel that explains it:
>
>
> https://www.intel.com/content/www/us/en/learn/what-is-a-real-time-system.html
>
> "In a soft real-time system, computers or equipment will continue to
> function after a missed deadline but may produce a lower-quality output.
> For example, latency in online video games can impact player interactions,
> but otherwise present no serious consequences."
>
> "Hard real-time systems have zero delay tolerance, and delayed signals can
> result in total failure or present immediate danger to users. Flight
> control systems and pacemakers are both examples where timeliness is not
> only essential but the lack of it can result in a life-or-death situation."
>
> I don't think it is inaccurate or misleading to call this mode real-time.
> It is soft real-time.
>
> On Thu, May 29, 2025 at 11:44 PM Mark Hamstra <ma...@gmail.com>
> wrote:
>
>> Clarifying what is meant by "real-time" and explicitly differentiating it
>> from actual real-time computing should be a bare minimum. I still don't
>> like the use of marketing-speak "real-time" that isn't really real-time in
>> engineering documents or API namespaces.
>>
>> On Thu, May 29, 2025 at 10:43 PM Jerry Peng <je...@gmail.com>
>> wrote:
>>
>>> Mark,
>>>
>>> I thought we are simply discussing the naming of the mode? Like I
>>> mentioned, if you think simply calling this mode "real-time" mode may cause
>>> confusion because "real-time" can mean other things in other fields, I can
>>> clarify what we mean by "real-time" explicitly in the SPIP document and any
>>> future documentation. That is not a problem and thank you for your feedback.
>>>
>>> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com>
>>> wrote:
>>>
>>>> Referencing other misuse of "real-time" is not persuasive. A SPIP is an
>>>> engineering document, not a marketing document. Technical clarity and
>>>> accuracy should be non-negotiable.
>>>>
>>>>
>>>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <
>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>
>>>>> Mark,
>>>>>
>>>>> As an example of my point if you go the the Apache Storm (another
>>>>> stream processing engine) website:
>>>>>
>>>>> https://storm.apache.org/
>>>>>
>>>>> It describes Storm as:
>>>>>
>>>>> "Apache Storm is a free and open source distributed *realtime*
>>>>> computation system"
>>>>>
>>>>> If you can to apache Flink:
>>>>>
>>>>>
>>>>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>>>>
>>>>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>>>>
>>>>> Thus, what the term "rea-time" implies in this should not be confusing
>>>>> for folks in this area.
>>>>>
>>>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <
>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>
>>>>>> Mich,
>>>>>>
>>>>>> If I understood your last email correctly, I think you also wanted to
>>>>>> have a discussion about naming? Why are we calling this new execution mode
>>>>>> described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
>>>>>> "continuous mode" is taken and we want another name to describe an
>>>>>> execution mode that provides ultra low latency processing. We could have
>>>>>> called it "low latency mode", though I don't really like that naming since
>>>>>> it implies the other execution modes are not low latency which I don't
>>>>>> believe is true. This new proposed mode can simply deliver even lower
>>>>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>>>>> are talking about "soft" real-time here. I think when we are talking about
>>>>>> distributed stream processing systems in the space of big data analytics,
>>>>>> it is reasonable to assume anything described in this space as "real-time"
>>>>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>>>>> can provide clear documentation on what "real-time" in real-time mode means
>>>>>> and what it guarantees. Just my thoughts. I would love to hear other
>>>>>> perspectives.
>>>>>>
>>>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> I think from what I have seen there are a good number of +1
>>>>>>> responses as opposed to quantitative discussions (based on my observations
>>>>>>> only). Given the objectives of the thread, we ought to focus on what is
>>>>>>> meant by real time compared to continuous modes.To be fair, it is a
>>>>>>> common point of confusion, and the terms are often used interchangeably in
>>>>>>> general conversation, but in technical contexts, especially with streaming
>>>>>>> data platforms, they have specific and important differences.
>>>>>>>
>>>>>>> "Continuous Mode" refers to a processing strategy that aims for
>>>>>>> true, uninterrupted, sub-millisecond latency processing. Chiefly
>>>>>>>
>>>>>>> - Event-at-a-Time (or very small batch groups): The system
>>>>>>> processes individual events or extremely small groups of events ->
>>>>>>> microbatches as they flow through the pipeline.
>>>>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>>>>> even below
>>>>>>> - Most business use cases (say financial markets) can live with
>>>>>>> this as they do not rely on rdges
>>>>>>>
>>>>>>> Now what is meant by "Real-time Mode"
>>>>>>>
>>>>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>>>>> that achieves real-time characteristics but might do so differently or more
>>>>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>>>>> mention, in real time application , there is nothing as an answer which is
>>>>>>> supposed to be late and correct. The timeliness is part of the application.
>>>>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>>>>> call the "Late and Correct is Useless" Principle
>>>>>>>
>>>>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>>>>> delivers low-latency processing with high reliability and ease of use,
>>>>>>> leveraging established, battle-tested components.I invite the audience to
>>>>>>> have a discussion on this.
>>>>>>>
>>>>>>> HTH
>>>>>>>
>>>>>>> Dr Mich Talebzadeh,
>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>>>>> > +1
>>>>>>>> >
>>>>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>>>>> >
>>>>>>>> > > +1.
>>>>>>>> > >
>>>>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com>
>>>>>>>> wrote:
>>>>>>>> > >
>>>>>>>> > >> +1
>>>>>>>> > >> Sent from my iPhone
>>>>>>>> > >>
>>>>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>>>>> wrote:
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >> +1 Nice feature
>>>>>>>> > >>
>>>>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>>>>> xyliyuanjian@gmail.com>
>>>>>>>> > >> wrote:
>>>>>>>> > >>
>>>>>>>> > >>> +1
>>>>>>>> > >>>
>>>>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>>>>> > >>>
>>>>>>>> > >>>> +1, LGTM.
>>>>>>>> > >>>>
>>>>>>>> > >>>> Kent
>>>>>>>> > >>>>
>>>>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>>>> > >>>>
>>>>>>>> > >>>>> +1. Super excited by this initiative!
>>>>>>>> > >>>>>
>>>>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>>>>> ybliang8@gmail.com>
>>>>>>>> > >>>>> wrote:
>>>>>>>> > >>>>>
>>>>>>>> > >>>>>> +1
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>>>>> huaxin.gao11@gmail.com>
>>>>>>>> > >>>>>> wrote:
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>>> +1
>>>>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we
>>>>>>>> can
>>>>>>>> > >>>>>>> eliminate the need for separate streaming engines,
>>>>>>>> reducing system
>>>>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>>>>> direction!
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>>> Hi,
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> My point about "in real time application or data, there
>>>>>>>> is nothing
>>>>>>>> > >>>>>>>> as an answer which is supposed to be late and correct.
>>>>>>>> The timeliness is
>>>>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>>>>> slowly it becomes
>>>>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we
>>>>>>>> need this
>>>>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of
>>>>>>>> the answer
>>>>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>>>>> streaming
>>>>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>>>>> delivers results that
>>>>>>>> > >>>>>>>> are technically "correct" but arrive too late to be
>>>>>>>> truly useful for
>>>>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes
>>>>>>>> them "useless or
>>>>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>>>>> *high-frequency
>>>>>>>> > >>>>>>>> trading,* market data or trade execution commands must be
>>>>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>>>>> mean missed
>>>>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>>>>> "correct" price
>>>>>>>> > >>>>>>>> update useless if it's not instantaneous. able for these
>>>>>>>> demanding
>>>>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply
>>>>>>>> not good enough. As
>>>>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>>>>> treated as such not
>>>>>>>> > >>>>>>>> as a comment.in SPIP
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>> Analysis |
>>>>>>>> > >>>>>>>> GDPR
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> view my Linkedin profile
>>>>>>>> > >>>>>>>> <
>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>>>>> denny.g.lee@gmail.com>
>>>>>>>> > >>>>>>>> wrote:
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>> Hey Mich,
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does
>>>>>>>> your
>>>>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>>>>>>>> comments directly
>>>>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet
>>>>>>>> below is a direct copy
>>>>>>>> > >>>>>>>>> from the SPIP itself.
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>> > >>>>>>>>> Denny
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>> just to add
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>>>>> definition of
>>>>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>>>>> application or
>>>>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed
>>>>>>>> to be late and
>>>>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if
>>>>>>>> I get the right
>>>>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>> Analysis |
>>>>>>>> > >>>>>>>>>> GDPR
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>>>>> > >>>>>>>>>> <
>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>>>>> micro-batching.If you
>>>>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction
>>>>>>>> must be balanced against
>>>>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>>>>> prevent back pressure
>>>>>>>> > >>>>>>>>>>> and instability. In the case of Continuous Processing
>>>>>>>> mode, a
>>>>>>>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>>>>>>>> interval quote
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> "
>>>>>>>> > >>>>>>>>>>> df.writeStream
>>>>>>>> > >>>>>>>>>>> .format("...")
>>>>>>>> > >>>>>>>>>>> .option("...")
>>>>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) //
>>>>>>>> new trigger
>>>>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>>>>> > >>>>>>>>>>> .start()
>>>>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should
>>>>>>>> run in the
>>>>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time
>>>>>>>> interval can also be
>>>>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long
>>>>>>>> each micro-batch should
>>>>>>>> > >>>>>>>>>>> run for.
>>>>>>>> > >>>>>>>>>>> "
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that
>>>>>>>> simple
>>>>>>>> > >>>>>>>>>>> HTH
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>> Analysis |
>>>>>>>> > >>>>>>>>>>> GDPR
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>>>>> > >>>>>>>>>>> <
>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> Hi all,
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP
>>>>>>>> titled
>>>>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured
>>>>>>>> Streaming” that I've been
>>>>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>>>>>>>> Jungtaek Lim, and
>>>>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>>>>> [Doc
>>>>>>>> > >>>>>>>>>>>> <
>>>>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>>>>> >
>>>>>>>> > >>>>>>>>>>>> ].
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>>>>> “Real-time Mode”
>>>>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly
>>>>>>>> lowers end-to-end latency
>>>>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility.
>>>>>>>> Our goal is
>>>>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs
>>>>>>>> that need results almost
>>>>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>>>>>>>> achieve this without
>>>>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>>>>> users already use – so
>>>>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>>>>> ultra-low-latency mode by
>>>>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>>>>> real-time
>>>>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>>>>> personalization) that
>>>>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>>>>> Spark’s current streaming
>>>>>>>> > >>>>>>>>>>>> engine.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> --
>>>>>>>> > >>>>>> Best,
>>>>>>>> > >>>>>> Yanbo
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>
>>>>>>>> > >>
>>>>>>>> > >> --
>>>>>>>> > >> John Zhuge
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Sakthi <sa...@apache.org>.
+1 (non-binding)
On Fri, May 30, 2025 at 2:39 PM Jules Damji <ju...@gmail.com> wrote:
> +1 (non-binding)
> —
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On May 30, 2025, at 12:39 PM, Mark Hamstra <ma...@gmail.com> wrote:
>
>
>
> A soft real-time system still defines an interval or frame within which
> results should be available, and often provides explicit warning or
> error-handling mechanisms when frame rates are missed. I see nothing like
> that in the SPIP. Instead, the length of the underlying microbatches is
> specified in the Trigger, but result reporting is just as quickly as
> possible with no reporting interval or frame rate specified and nothing
> that I can see happening if results take longer than the user is guessing
> or expecting. That's a low-latency, "we'll do it as fast as we can, but no
> promises or guarantees" system, not real-time.
>
> On Thu, May 29, 2025 at 11:57 PM Jerry Peng <je...@gmail.com>
> wrote:
>
>> Mark,
>>
>> For real-time systems there is a concept of "soft" real-time and "hard"
>> real-time systems. These concepts exist in textbooks. Here is a document
>> by intel that explains it:
>>
>>
>> https://www.intel.com/content/www/us/en/learn/what-is-a-real-time-system.html
>>
>> "In a soft real-time system, computers or equipment will continue to
>> function after a missed deadline but may produce a lower-quality output.
>> For example, latency in online video games can impact player interactions,
>> but otherwise present no serious consequences."
>>
>> "Hard real-time systems have zero delay tolerance, and delayed signals
>> can result in total failure or present immediate danger to users. Flight
>> control systems and pacemakers are both examples where timeliness is not
>> only essential but the lack of it can result in a life-or-death situation."
>>
>> I don't think it is inaccurate or misleading to call this mode
>> real-time. It is soft real-time.
>>
>> On Thu, May 29, 2025 at 11:44 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> Clarifying what is meant by "real-time" and explicitly differentiating
>>> it from actual real-time computing should be a bare minimum. I still don't
>>> like the use of marketing-speak "real-time" that isn't really real-time in
>>> engineering documents or API namespaces.
>>>
>>> On Thu, May 29, 2025 at 10:43 PM Jerry Peng <je...@gmail.com>
>>> wrote:
>>>
>>>> Mark,
>>>>
>>>> I thought we are simply discussing the naming of the mode? Like I
>>>> mentioned, if you think simply calling this mode "real-time" mode may cause
>>>> confusion because "real-time" can mean other things in other fields, I can
>>>> clarify what we mean by "real-time" explicitly in the SPIP document and any
>>>> future documentation. That is not a problem and thank you for your feedback.
>>>>
>>>> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Referencing other misuse of "real-time" is not persuasive. A SPIP is
>>>>> an engineering document, not a marketing document. Technical clarity and
>>>>> accuracy should be non-negotiable.
>>>>>
>>>>>
>>>>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <
>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>
>>>>>> Mark,
>>>>>>
>>>>>> As an example of my point if you go the the Apache Storm (another
>>>>>> stream processing engine) website:
>>>>>>
>>>>>> https://storm.apache.org/
>>>>>>
>>>>>> It describes Storm as:
>>>>>>
>>>>>> "Apache Storm is a free and open source distributed *realtime*
>>>>>> computation system"
>>>>>>
>>>>>> If you can to apache Flink:
>>>>>>
>>>>>>
>>>>>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>>>>>
>>>>>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>>>>>
>>>>>> Thus, what the term "rea-time" implies in this should not be
>>>>>> confusing for folks in this area.
>>>>>>
>>>>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <
>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>
>>>>>>> Mich,
>>>>>>>
>>>>>>> If I understood your last email correctly, I think you also wanted
>>>>>>> to have a discussion about naming? Why are we calling this new execution
>>>>>>> mode described in the SPIP "Real-time Mode"? Here are my two cents.
>>>>>>> Firstly, "continuous mode" is taken and we want another name to describe an
>>>>>>> execution mode that provides ultra low latency processing. We could have
>>>>>>> called it "low latency mode", though I don't really like that naming since
>>>>>>> it implies the other execution modes are not low latency which I don't
>>>>>>> believe is true. This new proposed mode can simply deliver even lower
>>>>>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>>>>>> are talking about "soft" real-time here. I think when we are talking about
>>>>>>> distributed stream processing systems in the space of big data analytics,
>>>>>>> it is reasonable to assume anything described in this space as "real-time"
>>>>>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>>>>>> can provide clear documentation on what "real-time" in real-time mode means
>>>>>>> and what it guarantees. Just my thoughts. I would love to hear other
>>>>>>> perspectives.
>>>>>>>
>>>>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>
>>>>>>>> I think from what I have seen there are a good number of +1
>>>>>>>> responses as opposed to quantitative discussions (based on my observations
>>>>>>>> only). Given the objectives of the thread, we ought to focus on what is
>>>>>>>> meant by real time compared to continuous modes.To be fair, it is a
>>>>>>>> common point of confusion, and the terms are often used interchangeably in
>>>>>>>> general conversation, but in technical contexts, especially with streaming
>>>>>>>> data platforms, they have specific and important differences.
>>>>>>>>
>>>>>>>> "Continuous Mode" refers to a processing strategy that aims for
>>>>>>>> true, uninterrupted, sub-millisecond latency processing. Chiefly
>>>>>>>>
>>>>>>>> - Event-at-a-Time (or very small batch groups): The system
>>>>>>>> processes individual events or extremely small groups of events ->
>>>>>>>> microbatches as they flow through the pipeline.
>>>>>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>>>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>>>>>> even below
>>>>>>>> - Most business use cases (say financial markets) can live with
>>>>>>>> this as they do not rely on rdges
>>>>>>>>
>>>>>>>> Now what is meant by "Real-time Mode"
>>>>>>>>
>>>>>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>>>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>>>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>>>>>> that achieves real-time characteristics but might do so differently or more
>>>>>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>>>>>> mention, in real time application , there is nothing as an answer which is
>>>>>>>> supposed to be late and correct. The timeliness is part of the application.
>>>>>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>>>>>> call the "Late and Correct is Useless" Principle
>>>>>>>>
>>>>>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>>>>>> delivers low-latency processing with high reliability and ease of use,
>>>>>>>> leveraging established, battle-tested components.I invite the audience to
>>>>>>>> have a discussion on this.
>>>>>>>>
>>>>>>>> HTH
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>> GDPR
>>>>>>>>
>>>>>>>> view my Linkedin profile
>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>>>>>> > +1
>>>>>>>>> >
>>>>>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>>>>>> >
>>>>>>>>> > > +1.
>>>>>>>>> > >
>>>>>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com>
>>>>>>>>> wrote:
>>>>>>>>> > >
>>>>>>>>> > >> +1
>>>>>>>>> > >> Sent from my iPhone
>>>>>>>>> > >>
>>>>>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >> +1 Nice feature
>>>>>>>>> > >>
>>>>>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>>>>>> xyliyuanjian@gmail.com>
>>>>>>>>> > >> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>> +1
>>>>>>>>> > >>>
>>>>>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>>>>>> > >>>
>>>>>>>>> > >>>> +1, LGTM.
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> Kent
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>>>>> > >>>>
>>>>>>>>> > >>>>> +1. Super excited by this initiative!
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>>>>>> ybliang8@gmail.com>
>>>>>>>>> > >>>>> wrote:
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>>> +1
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>>>>>> huaxin.gao11@gmail.com>
>>>>>>>>> > >>>>>> wrote:
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>>> +1
>>>>>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we
>>>>>>>>> can
>>>>>>>>> > >>>>>>> eliminate the need for separate streaming engines,
>>>>>>>>> reducing system
>>>>>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>>>>>> direction!
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>>> Hi,
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> My point about "in real time application or data, there
>>>>>>>>> is nothing
>>>>>>>>> > >>>>>>>> as an answer which is supposed to be late and correct.
>>>>>>>>> The timeliness is
>>>>>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>>>>>> slowly it becomes
>>>>>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we
>>>>>>>>> need this
>>>>>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of
>>>>>>>>> the answer
>>>>>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>>>>>> streaming
>>>>>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>>>>>> delivers results that
>>>>>>>>> > >>>>>>>> are technically "correct" but arrive too late to be
>>>>>>>>> truly useful for
>>>>>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes
>>>>>>>>> them "useless or
>>>>>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>>>>>> *high-frequency
>>>>>>>>> > >>>>>>>> trading,* market data or trade execution commands must
>>>>>>>>> be
>>>>>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>>>>>> mean missed
>>>>>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>>>>>> "correct" price
>>>>>>>>> > >>>>>>>> update useless if it's not instantaneous. able for
>>>>>>>>> these demanding
>>>>>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply
>>>>>>>>> not good enough. As
>>>>>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>>>>>> treated as such not
>>>>>>>>> > >>>>>>>> as a comment.in SPIP
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>>> Analysis |
>>>>>>>>> > >>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>>>>>> denny.g.lee@gmail.com>
>>>>>>>>> > >>>>>>>> wrote:
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>> Hey Mich,
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does
>>>>>>>>> your
>>>>>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps
>>>>>>>>> add comments directly
>>>>>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet
>>>>>>>>> below is a direct copy
>>>>>>>>> > >>>>>>>>> from the SPIP itself.
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>> > >>>>>>>>> Denny
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>> just to add
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>>>>>> definition of
>>>>>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>>>>>> application or
>>>>>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed
>>>>>>>>> to be late and
>>>>>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if
>>>>>>>>> I get the right
>>>>>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>>> Analysis |
>>>>>>>>> > >>>>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>>>>>> micro-batching.If you
>>>>>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction
>>>>>>>>> must be balanced against
>>>>>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>>>>>> prevent back pressure
>>>>>>>>> > >>>>>>>>>>> and instability. In the case of Continuous
>>>>>>>>> Processing mode, a
>>>>>>>>> > >>>>>>>>>>> specific continuous trigger with a desired
>>>>>>>>> checkpoint interval quote
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> "
>>>>>>>>> > >>>>>>>>>>> df.writeStream
>>>>>>>>> > >>>>>>>>>>> .format("...")
>>>>>>>>> > >>>>>>>>>>> .option("...")
>>>>>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) //
>>>>>>>>> new trigger
>>>>>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>>>>>> > >>>>>>>>>>> .start()
>>>>>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should
>>>>>>>>> run in the
>>>>>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time
>>>>>>>>> interval can also be
>>>>>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long
>>>>>>>>> each micro-batch should
>>>>>>>>> > >>>>>>>>>>> run for.
>>>>>>>>> > >>>>>>>>>>> "
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that
>>>>>>>>> simple
>>>>>>>>> > >>>>>>>>>>> HTH
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime |
>>>>>>>>> Forensic Analysis |
>>>>>>>>> > >>>>>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> Hi all,
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP
>>>>>>>>> titled
>>>>>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured
>>>>>>>>> Streaming” that I've been
>>>>>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao
>>>>>>>>> Sun, Jungtaek Lim, and
>>>>>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>>>>>> [Doc
>>>>>>>>> > >>>>>>>>>>>> <
>>>>>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>>>>>> >
>>>>>>>>> > >>>>>>>>>>>> ].
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>>>>>> “Real-time Mode”
>>>>>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly
>>>>>>>>> lowers end-to-end latency
>>>>>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility.
>>>>>>>>> Our goal is
>>>>>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs
>>>>>>>>> that need results almost
>>>>>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want
>>>>>>>>> to achieve this without
>>>>>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>>>>>> users already use – so
>>>>>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>>>>>> ultra-low-latency mode by
>>>>>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>>>>>> real-time
>>>>>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>>>>>> personalization) that
>>>>>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>>>>>> Spark’s current streaming
>>>>>>>>> > >>>>>>>>>>>> engine.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> --
>>>>>>>>> > >>>>>> Best,
>>>>>>>>> > >>>>>> Yanbo
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>
>>>>>>>>> > >>
>>>>>>>>> > >> --
>>>>>>>>> > >> John Zhuge
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>
>>>>>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jules Damji <ju...@gmail.com>.
+1 (non-binding)
—
Sent from my iPhone
Pardon the dumb thumb typos :)
> On May 30, 2025, at 12:39 PM, Mark Hamstra <ma...@gmail.com> wrote:
>
>
>
>
> A soft real-time system still defines an interval or frame within which
> results should be available, and often provides explicit warning or error-
> handling mechanisms when frame rates are missed. I see nothing like that in
> the SPIP. Instead, the length of the underlying microbatches is specified in
> the Trigger, but result reporting is just as quickly as possible with no
> reporting interval or frame rate specified and nothing that I can see
> happening if results take longer than the user is guessing or expecting.
> That's a low-latency, "we'll do it as fast as we can, but no promises or
> guarantees" system, not real-time.
>
>
>
>
> On Thu, May 29, 2025 at 11:57 PM Jerry Peng
> <[jerry.boyang.peng@gmail.com](mailto:jerry.boyang.peng@gmail.com)> wrote:
>
>
>> Mark,
>
> For real-time systems there is a concept of "soft" real-time and "hard"
> real-time systems. These concepts exist in textbooks. Here is a document
> by intel that explains it:
>>
>>
>
>>
>> <https://www.intel.com/content/www/us/en/learn/what-is-a-real-time-
system.html>
>
> "In a soft real-time system, computers or equipment will continue to
> function after a missed deadline but may produce a lower-quality output. For
> example, latency in online video games can impact player interactions, but
> otherwise present no serious consequences."
>
> "Hard real-time systems have zero delay tolerance, and delayed signals can
> result in total failure or present immediate danger to users. Flight control
> systems and pacemakers are both examples where timeliness is not only
> essential but the lack of it can result in a life-or-death situation."
>>
>>
>
>>
>> I don't think it is inaccurate or misleading to call this mode real-time.
It is soft real-time.
>>
>>
>
>>
>> On Thu, May 29, 2025 at 11:44 PM Mark Hamstra
<[markhamstra@gmail.com](mailto:markhamstra@gmail.com)> wrote:
>
>>
>>> Clarifying what is meant by "real-time" and explicitly differentiating it
from actual real-time computing should be a bare minimum. I still don't like
the use of marketing-speak "real-time" that isn't really real-time in
engineering documents or API namespaces.
>>>
>>>
>
>>>
>>> On Thu, May 29, 2025 at 10:43 PM Jerry Peng
<[jerry.boyang.peng@gmail.com](mailto:jerry.boyang.peng@gmail.com)> wrote:
>
>>>
>>>> Mark,
>>>>
>>>>
>
>>>>
>>>> I thought we are simply discussing the naming of the mode? Like I
mentioned, if you think simply calling this mode "real-time" mode may cause
confusion because "real-time" can mean other things in other fields, I can
clarify what we mean by "real-time" explicitly in the SPIP document and any
future documentation. That is not a problem and thank you for your feedback.
>>>>
>>>>
>
>>>>
>>>> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra
<[markhamstra@gmail.com](mailto:markhamstra@gmail.com)> wrote:
>
>>>>
>>>>> Referencing other misuse of "real-time" is not persuasive. A SPIP is an
engineering document, not a marketing document. Technical clarity and accuracy
should be non-negotiable.
>>>>>
>>>>>
>
>>>>>
>>>>>
>
>>>>>
>>>>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng
<[jerry.boyang.peng@gmail.com](mailto:jerry.boyang.peng@gmail.com)> wrote:
>
>>>>>
>>>>>> Mark,
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> As an example of my point if you go the the Apache Storm (another
stream processing engine) website:
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> <https://storm.apache.org/>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> It describes Storm as:
>
> "Apache Storm is a free and open source distributed **realtime** computation
> system"
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> If you can to apache Flink:
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> <https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-
real-time-data-processing/>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> "Apache Flink 2.0.0: A new Era of **Real-Time** Data Processing"
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> Thus, what the term "rea-time" implies in this should not be confusing
for folks in this area.
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng
<[jerry.boyang.peng@gmail.com](mailto:jerry.boyang.peng@gmail.com)> wrote:
>
>>>>>>
>>>>>>> Mich,
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> If I understood your last email correctly, I think you also wanted to
have a discussion about naming? Why are we calling this new execution mode
described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
"continuous mode" is taken and we want another name to describe an execution
mode that provides ultra low latency processing. We could have called it "low
latency mode", though I don't really like that naming since it implies the
other execution modes are not low latency which I don't believe is true. This
new proposed mode can simply deliver even lower latency. Thus, we came up
with the name "Real-time Mode". Of course, we are talking about "soft" real-
time here. I think when we are talking about distributed stream processing
systems in the space of big data analytics, it is reasonable to assume
anything described in this space as "real-time" implies "soft" real-time.
Though if this is confusing or misleading, we can provide clear documentation
on what "real-time" in real-time mode means and what it guarantees. Just my
thoughts. I would love to hear other perspectives.
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh
<[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:
>
>>>>>>>
>>>>>>>> I think from what I have seen there are a good number of +1 responses
as opposed to quantitative discussions (based on my observations only). Given
the objectives of the thread, we ought to focus on what is meant by real time
compared to continuous modes.To be fair, it is a common point of confusion,
and the terms are often used interchangeably in general conversation, but in
technical contexts, especially with streaming data platforms, they have
specific and important differences.
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> "Continuous Mode" refers to a processing strategy that aims for true,
uninterrupted, sub-millisecond latency processing. Chiefly
>
>>>>>>>>
>>>>>>>> * Event-at-a-Time (or very small batch groups): The system
processes individual events or extremely small groups of events ->
microbatches as they flow through the pipeline.
>>>>>>>> * Minimal Latency: The primary goal is to achieve the absolute
lowest possible end-to-end latency, often in the order of milliseconds or even
below
>>>>>>>> * Most business use cases (say financial markets) can live with
this as they do not rely on rdges
>>>>>>>>
Now what is meant by "Real-time Mode"
>
> This is where the nuance comes in. "Real-time" is a broader and sometimes
> more subjective term. When the text introduces "Real-time Mode" as distinct
> from "Continuous Mode," it suggests a specific implementation that achieves
> real-time characteristics but might do so differently or more robustly than
> a "continuous" mode attempt. Going back to my earlier mention, in real time
> application , there is nothing as an answer which is supposed to be late and
> correct. The timeliness is part of the application. if I get the right
> answer too slowly it becomes useless or wrong. What I call the "Late and
> Correct is Useless" Principle
>
> In summary, "Real-time Mode" seems to describe an approach that delivers
> low-latency processing with high reliability and ease of use, leveraging
> established, battle-tested components.I invite the audience to have a
> discussion on this.
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> HTH
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> 
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie
<[yangjie01@apache.org](mailto:yangjie01@apache.org)> wrote:
>
>>>>>>>>
>>>>>>>>> +1
>
> On 2025/05/29 16:25:19 Xiao Li wrote:
> > +1
> >
> > Yuming Wang <[yumwang@apache.org](mailto:yumwang@apache.org)>
> 于2025年5月29日周四 02:22写道：
> >
> > > +1.
> > >
> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai
> <[dbtsai@dbtsai.com](mailto:dbtsai@dbtsai.com)> wrote:
> > >
> > >> +1
> > >> Sent from my iPhone
> > >>
> > >> On May 29, 2025, at 12:15 AM, John Zhuge
> <[jzhuge@apache.org](mailto:jzhuge@apache.org)> wrote:
> > >>
> > >>
> > >> +1 Nice feature
> > >>
> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li
> <[xyliyuanjian@gmail.com](mailto:xyliyuanjian@gmail.com)>
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>> Kent Yao <[yao@apache.org](mailto:yao@apache.org)> 于2025年5月28日周三
> 19:31写道：
> > >>>
> > >>>> +1, LGTM.
> > >>>>
> > >>>> Kent
> > >>>>
> > >>>> 在 2025年5月29日星期四，Chao Sun
> <[sunchao@apache.org](mailto:sunchao@apache.org)> 写道：
> > >>>>
> > >>>>> +1. Super excited by this initiative!
> > >>>>>
> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang
> <[ybliang8@gmail.com](mailto:ybliang8@gmail.com)>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> +1
> > >>>>>>
> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao
> <[huaxin.gao11@gmail.com](mailto:huaxin.gao11@gmail.com)>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> +1
> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
> > >>>>>>> eliminate the need for separate streaming engines, reducing
> system
> > >>>>>>> complexity and operational cost. Excited to see this direction!
> > >>>>>>>
> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
> > >>>>>>> [mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)>
> wrote:
> > >>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> My point about "in real time application or data, there is
> nothing
> > >>>>>>>> as an answer which is supposed to be late and correct. The
> timeliness is
> > >>>>>>>> part of the application. if I get the right answer too slowly it
> becomes
> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need this
> > >>>>>>>> Spark Structured Streaming proposal.
> > >>>>>>>>
> > >>>>>>>> The proposal is precisely about enabling Spark to power
> > >>>>>>>> applications where, as I define it, the *timeliness* of the
> answer
> > >>>>>>>> is as critical as its *correctness*. Spark's current streaming
> > >>>>>>>> engine, primarily operating on micro-batches, often delivers
> results that
> > >>>>>>>> are technically "correct" but arrive too late to be truly useful
> for
> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
> "useless or
> > >>>>>>>> wrong" in a practical, business-critical sense.
> > >>>>>>>>
> > >>>>>>>> For example *in real-time fraud detection* and In *high-
> frequency
> > >>>>>>>> trading,* market data or trade execution commands must be
> > >>>>>>>> delivered with minimal latency. Even a slight delay can mean
> missed
> > >>>>>>>> opportunities or significant financial losses, making a
> "correct" price
> > >>>>>>>> update useless if it's not instantaneous. able for these
> demanding
> > >>>>>>>> use cases, where a "late but correct" answer is simply not good
> enough. As
> > >>>>>>>> a colliery it is a fundamental concept, so it has to be treated
> as such not
> > >>>>>>>> as a [comment.in](http://comment.in) SPIP
> > >>>>>>>>
> > >>>>>>>> Hope this clarifies the connection in practical terms
> > >>>>>>>> Dr Mich Talebzadeh,
> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
> > >>>>>>>> GDPR
> > >>>>>>>>
> > >>>>>>>> view my Linkedin profile
> > >>>>>>>> <<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee
> <[denny.g.lee@gmail.com](mailto:denny.g.lee@gmail.com)>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hey Mich,
> > >>>>>>>>>
> > >>>>>>>>> Sorry, I may be missing something here but what does your
> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
> comments directly
> > >>>>>>>>> to the SPIP to provide context as the code snippet below is a
> direct copy
> > >>>>>>>>> from the SPIP itself.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Denny
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
> > >>>>>>>>> [mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)>
> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> just to add
> > >>>>>>>>>>
> > >>>>>>>>>> A stronger definition of real time. The engineering definition
> of
> > >>>>>>>>>> real time is roughly fast enough to be interactive
> > >>>>>>>>>>
> > >>>>>>>>>> However, I put a stronger definition. In real time application
> or
> > >>>>>>>>>> data, there is nothing as an answer which is supposed to be
> late and
> > >>>>>>>>>> correct. The timeliness is part of the application.if I get
> the right
> > >>>>>>>>>> answer too slowly it becomes useless or wrong
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Dr Mich Talebzadeh,
> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis
> |
> > >>>>>>>>>> GDPR
> > >>>>>>>>>>
> > >>>>>>>>>> view my Linkedin profile
> > >>>>>>>>>> <<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
> > >>>>>>>>>> [mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)>
> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> The current limitations in SSS come from micro-batching.If
> you
> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must be
> balanced against
> > >>>>>>>>>>> the available processing capacity of the cluster to prevent
> back pressure
> > >>>>>>>>>>> and instability. In the case of Continuous Processing mode, a
> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
> interval quote
> > >>>>>>>>>>>
> > >>>>>>>>>>> "
> > >>>>>>>>>>> df.writeStream
> > >>>>>>>>>>> .format("...")
> > >>>>>>>>>>> .option("...")
> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
> trigger
> > >>>>>>>>>>> type to enable real-time Mode
> > >>>>>>>>>>> .start()
> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run in
> the
> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval can
> also be
> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each
> micro-batch should
> > >>>>>>>>>>> run for.
> > >>>>>>>>>>> "
> > >>>>>>>>>>>
> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
> > >>>>>>>>>>> HTH
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Dr Mich Talebzadeh,
> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
> Analysis |
> > >>>>>>>>>>> GDPR
> > >>>>>>>>>>>
> > >>>>>>>>>>> view my Linkedin profile
> > >>>>>>>>>>> <<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
> > >>>>>>>>>>>
> [jerry.boyang.peng@gmail.com](mailto:jerry.boyang.peng@gmail.com)> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that
> I've been
> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
> Jungtaek Lim, and
> > >>>>>>>>>>>> Michael Armbrust: [JIRA
> > >>>>>>>>>>>> <<https://issues.apache.org/jira/browse/SPARK-52330>>] [Doc
> > >>>>>>>>>>>>
> <<https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>>
> > >>>>>>>>>>>> ].
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time
> Mode”
> > >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers end-
> to-end latency
> > >>>>>>>>>>>> for processing streams of data.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our goal
> is
> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that need
> results almost
> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve
> this without
> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users
> already use – so
> > >>>>>>>>>>>> existing streaming queries can run in this new ultra-low-
> latency mode by
> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
> personalization) that
> > >>>>>>>>>>>> today cannot meet their latency requirements with Spark’s
> current streaming
> > >>>>>>>>>>>> engine.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
> > >>>>>>>>>>>> suggestions on this approach!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>
> > >>>>>> \--
> > >>>>>> Best,
> > >>>>>> Yanbo
> > >>>>>>
> > >>>>>
> > >>
> > >> \--
> > >> John Zhuge
> > >>
> > >>
> >
>
> \---------------------------------------------------------------------
> To unsubscribe e-mail: [dev-unsubscribe@spark.apache.org](mailto:dev-
> unsubscribe@spark.apache.org)
>
>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mich Talebzadeh <mi...@gmail.com>.
ok fair points
This SPIP (Structured Streaming, in this context) admittedly does not meet
the rigorous, academic definition of a soft real-time system, due to the
lack of explicit, guaranteed deadlines and internal mechanisms for handling
missed frames.
Having said that, despite not being a "strict" soft real-time system, the
SPIP offers significant advantages that make it highly valuable for many
real-world applications:
- Lower Latency: Much faster than traditional batch processing.
- High Throughput & Scalability: Handles large volumes of data
effectively.
- Robust Fault Tolerance: Reliable operation even with failures.
- Ease of Integration: Fits well into existing data architectures
without significant re-writes
"Real-time Enough" for Many: In other words for a vast number of business
use cases, where delays are undesirable but not catastrophic (i.e., not
life-or-death or causing immediate, irreparable damage), the SPIP's
performance is sufficient. This addresses the practical reality of how
"real-time" is used in the industry.
Proposed Better Terminology -> One can propose for more precise terms like
"near real-time streaming" or "interactive streaming" to accurately
describe the system's capabilities and bridge the gap between academic
rigor and practical industry usage. This IMO is a good suggestion to reduce
ambiguity.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Fri, 30 May 2025 at 20:39, Mark Hamstra <ma...@gmail.com> wrote:
> A soft real-time system still defines an interval or frame within which
> results should be available, and often provides explicit warning or
> error-handling mechanisms when frame rates are missed. I see nothing like
> that in the SPIP. Instead, the length of the underlying microbatches is
> specified in the Trigger, but result reporting is just as quickly as
> possible with no reporting interval or frame rate specified and nothing
> that I can see happening if results take longer than the user is guessing
> or expecting. That's a low-latency, "we'll do it as fast as we can, but no
> promises or guarantees" system, not real-time.
>
> On Thu, May 29, 2025 at 11:57 PM Jerry Peng <je...@gmail.com>
> wrote:
>
>> Mark,
>>
>> For real-time systems there is a concept of "soft" real-time and "hard"
>> real-time systems. These concepts exist in textbooks. Here is a document
>> by intel that explains it:
>>
>>
>> https://www.intel.com/content/www/us/en/learn/what-is-a-real-time-system.html
>>
>> "In a soft real-time system, computers or equipment will continue to
>> function after a missed deadline but may produce a lower-quality output.
>> For example, latency in online video games can impact player interactions,
>> but otherwise present no serious consequences."
>>
>> "Hard real-time systems have zero delay tolerance, and delayed signals
>> can result in total failure or present immediate danger to users. Flight
>> control systems and pacemakers are both examples where timeliness is not
>> only essential but the lack of it can result in a life-or-death situation."
>>
>> I don't think it is inaccurate or misleading to call this mode
>> real-time. It is soft real-time.
>>
>> On Thu, May 29, 2025 at 11:44 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> Clarifying what is meant by "real-time" and explicitly differentiating
>>> it from actual real-time computing should be a bare minimum. I still don't
>>> like the use of marketing-speak "real-time" that isn't really real-time in
>>> engineering documents or API namespaces.
>>>
>>> On Thu, May 29, 2025 at 10:43 PM Jerry Peng <je...@gmail.com>
>>> wrote:
>>>
>>>> Mark,
>>>>
>>>> I thought we are simply discussing the naming of the mode? Like I
>>>> mentioned, if you think simply calling this mode "real-time" mode may cause
>>>> confusion because "real-time" can mean other things in other fields, I can
>>>> clarify what we mean by "real-time" explicitly in the SPIP document and any
>>>> future documentation. That is not a problem and thank you for your feedback.
>>>>
>>>> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Referencing other misuse of "real-time" is not persuasive. A SPIP is
>>>>> an engineering document, not a marketing document. Technical clarity and
>>>>> accuracy should be non-negotiable.
>>>>>
>>>>>
>>>>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <
>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>
>>>>>> Mark,
>>>>>>
>>>>>> As an example of my point if you go the the Apache Storm (another
>>>>>> stream processing engine) website:
>>>>>>
>>>>>> https://storm.apache.org/
>>>>>>
>>>>>> It describes Storm as:
>>>>>>
>>>>>> "Apache Storm is a free and open source distributed *realtime*
>>>>>> computation system"
>>>>>>
>>>>>> If you can to apache Flink:
>>>>>>
>>>>>>
>>>>>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>>>>>
>>>>>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>>>>>
>>>>>> Thus, what the term "rea-time" implies in this should not be
>>>>>> confusing for folks in this area.
>>>>>>
>>>>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <
>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>
>>>>>>> Mich,
>>>>>>>
>>>>>>> If I understood your last email correctly, I think you also wanted
>>>>>>> to have a discussion about naming? Why are we calling this new execution
>>>>>>> mode described in the SPIP "Real-time Mode"? Here are my two cents.
>>>>>>> Firstly, "continuous mode" is taken and we want another name to describe an
>>>>>>> execution mode that provides ultra low latency processing. We could have
>>>>>>> called it "low latency mode", though I don't really like that naming since
>>>>>>> it implies the other execution modes are not low latency which I don't
>>>>>>> believe is true. This new proposed mode can simply deliver even lower
>>>>>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>>>>>> are talking about "soft" real-time here. I think when we are talking about
>>>>>>> distributed stream processing systems in the space of big data analytics,
>>>>>>> it is reasonable to assume anything described in this space as "real-time"
>>>>>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>>>>>> can provide clear documentation on what "real-time" in real-time mode means
>>>>>>> and what it guarantees. Just my thoughts. I would love to hear other
>>>>>>> perspectives.
>>>>>>>
>>>>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>
>>>>>>>> I think from what I have seen there are a good number of +1
>>>>>>>> responses as opposed to quantitative discussions (based on my observations
>>>>>>>> only). Given the objectives of the thread, we ought to focus on what is
>>>>>>>> meant by real time compared to continuous modes.To be fair, it is a
>>>>>>>> common point of confusion, and the terms are often used interchangeably in
>>>>>>>> general conversation, but in technical contexts, especially with streaming
>>>>>>>> data platforms, they have specific and important differences.
>>>>>>>>
>>>>>>>> "Continuous Mode" refers to a processing strategy that aims for
>>>>>>>> true, uninterrupted, sub-millisecond latency processing. Chiefly
>>>>>>>>
>>>>>>>> - Event-at-a-Time (or very small batch groups): The system
>>>>>>>> processes individual events or extremely small groups of events ->
>>>>>>>> microbatches as they flow through the pipeline.
>>>>>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>>>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>>>>>> even below
>>>>>>>> - Most business use cases (say financial markets) can live with
>>>>>>>> this as they do not rely on rdges
>>>>>>>>
>>>>>>>> Now what is meant by "Real-time Mode"
>>>>>>>>
>>>>>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>>>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>>>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>>>>>> that achieves real-time characteristics but might do so differently or more
>>>>>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>>>>>> mention, in real time application , there is nothing as an answer which is
>>>>>>>> supposed to be late and correct. The timeliness is part of the application.
>>>>>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>>>>>> call the "Late and Correct is Useless" Principle
>>>>>>>>
>>>>>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>>>>>> delivers low-latency processing with high reliability and ease of use,
>>>>>>>> leveraging established, battle-tested components.I invite the audience to
>>>>>>>> have a discussion on this.
>>>>>>>>
>>>>>>>> HTH
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>> GDPR
>>>>>>>>
>>>>>>>> view my Linkedin profile
>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>>>>>> > +1
>>>>>>>>> >
>>>>>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>>>>>> >
>>>>>>>>> > > +1.
>>>>>>>>> > >
>>>>>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com>
>>>>>>>>> wrote:
>>>>>>>>> > >
>>>>>>>>> > >> +1
>>>>>>>>> > >> Sent from my iPhone
>>>>>>>>> > >>
>>>>>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >> +1 Nice feature
>>>>>>>>> > >>
>>>>>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>>>>>> xyliyuanjian@gmail.com>
>>>>>>>>> > >> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>> +1
>>>>>>>>> > >>>
>>>>>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>>>>>> > >>>
>>>>>>>>> > >>>> +1, LGTM.
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> Kent
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>>>>> > >>>>
>>>>>>>>> > >>>>> +1. Super excited by this initiative!
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>>>>>> ybliang8@gmail.com>
>>>>>>>>> > >>>>> wrote:
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>>> +1
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>>>>>> huaxin.gao11@gmail.com>
>>>>>>>>> > >>>>>> wrote:
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>>> +1
>>>>>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we
>>>>>>>>> can
>>>>>>>>> > >>>>>>> eliminate the need for separate streaming engines,
>>>>>>>>> reducing system
>>>>>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>>>>>> direction!
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>>> Hi,
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> My point about "in real time application or data, there
>>>>>>>>> is nothing
>>>>>>>>> > >>>>>>>> as an answer which is supposed to be late and correct.
>>>>>>>>> The timeliness is
>>>>>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>>>>>> slowly it becomes
>>>>>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we
>>>>>>>>> need this
>>>>>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of
>>>>>>>>> the answer
>>>>>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>>>>>> streaming
>>>>>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>>>>>> delivers results that
>>>>>>>>> > >>>>>>>> are technically "correct" but arrive too late to be
>>>>>>>>> truly useful for
>>>>>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes
>>>>>>>>> them "useless or
>>>>>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>>>>>> *high-frequency
>>>>>>>>> > >>>>>>>> trading,* market data or trade execution commands must
>>>>>>>>> be
>>>>>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>>>>>> mean missed
>>>>>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>>>>>> "correct" price
>>>>>>>>> > >>>>>>>> update useless if it's not instantaneous. able for
>>>>>>>>> these demanding
>>>>>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply
>>>>>>>>> not good enough. As
>>>>>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>>>>>> treated as such not
>>>>>>>>> > >>>>>>>> as a comment.in SPIP
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>>> Analysis |
>>>>>>>>> > >>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>>>>>> denny.g.lee@gmail.com>
>>>>>>>>> > >>>>>>>> wrote:
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>> Hey Mich,
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does
>>>>>>>>> your
>>>>>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps
>>>>>>>>> add comments directly
>>>>>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet
>>>>>>>>> below is a direct copy
>>>>>>>>> > >>>>>>>>> from the SPIP itself.
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>> > >>>>>>>>> Denny
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>> just to add
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>>>>>> definition of
>>>>>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>>>>>> application or
>>>>>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed
>>>>>>>>> to be late and
>>>>>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if
>>>>>>>>> I get the right
>>>>>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>>> Analysis |
>>>>>>>>> > >>>>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>>>>>> micro-batching.If you
>>>>>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction
>>>>>>>>> must be balanced against
>>>>>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>>>>>> prevent back pressure
>>>>>>>>> > >>>>>>>>>>> and instability. In the case of Continuous
>>>>>>>>> Processing mode, a
>>>>>>>>> > >>>>>>>>>>> specific continuous trigger with a desired
>>>>>>>>> checkpoint interval quote
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> "
>>>>>>>>> > >>>>>>>>>>> df.writeStream
>>>>>>>>> > >>>>>>>>>>> .format("...")
>>>>>>>>> > >>>>>>>>>>> .option("...")
>>>>>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) //
>>>>>>>>> new trigger
>>>>>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>>>>>> > >>>>>>>>>>> .start()
>>>>>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should
>>>>>>>>> run in the
>>>>>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time
>>>>>>>>> interval can also be
>>>>>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long
>>>>>>>>> each micro-batch should
>>>>>>>>> > >>>>>>>>>>> run for.
>>>>>>>>> > >>>>>>>>>>> "
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that
>>>>>>>>> simple
>>>>>>>>> > >>>>>>>>>>> HTH
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime |
>>>>>>>>> Forensic Analysis |
>>>>>>>>> > >>>>>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> Hi all,
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP
>>>>>>>>> titled
>>>>>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured
>>>>>>>>> Streaming” that I've been
>>>>>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao
>>>>>>>>> Sun, Jungtaek Lim, and
>>>>>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>>>>>> [Doc
>>>>>>>>> > >>>>>>>>>>>> <
>>>>>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>>>>>> >
>>>>>>>>> > >>>>>>>>>>>> ].
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>>>>>> “Real-time Mode”
>>>>>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly
>>>>>>>>> lowers end-to-end latency
>>>>>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility.
>>>>>>>>> Our goal is
>>>>>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs
>>>>>>>>> that need results almost
>>>>>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want
>>>>>>>>> to achieve this without
>>>>>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>>>>>> users already use – so
>>>>>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>>>>>> ultra-low-latency mode by
>>>>>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>>>>>> real-time
>>>>>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>>>>>> personalization) that
>>>>>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>>>>>> Spark’s current streaming
>>>>>>>>> > >>>>>>>>>>>> engine.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> --
>>>>>>>>> > >>>>>> Best,
>>>>>>>>> > >>>>>> Yanbo
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>
>>>>>>>>> > >>
>>>>>>>>> > >> --
>>>>>>>>> > >> John Zhuge
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>
>>>>>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jungtaek Lim <ka...@gmail.com>.
I agree and I appreciate your input to clarify the term and the gap we have
from the theoretical definition.
I just would like to put some color here for just 2 cents.
It is not uncommon for the technical term to be re-interpreted and
expanded. One of the known examples is "exactly-once processing" semantics.
Does this mean the engine processes the data only once? No. Most streaming
engines deal with this via loosening the definition to be "effectively
once" (the term was mentioned in Jay Krep's relevant blog post
<https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f>),
via ensuring at-least-once processing semantic plus making the writes to be
idempotent or be transactional. That said, upon failure, the engine
processes the data multiple times (at-least-once), but the output will be
visible for the same when the engine processes the data as
true exactly-once semantic.
So does the industry try to correct the term? No, because it is still
practically correct in some aspects. If someone implies the strict
definition of exactly-once to build their streaming app, they will fail to
see their own expectation upon failure. The checkpoint interval is
important in the restoration of the streaming app because there is an
implication that we reprocess data, which breaks the expectation in some
sense. Still, the term "exactly-once" is used, because everyone's focus is
about duplications of the output and it fits that aspect.
That said, the thing is how moderate users would reason about the term
"real-time" on the streaming engine. Likewise Jerry referred to two
well-known open source streaming engines, the term "real-time" was not used
strictly to represent the definition in the textbook. That has been there
for more than 10 years. I have to speculate, but the term "real-time" is
probably leveraged for emphasizing the difference with batch query/app.
Is 'real-time' a marketing term? Probably yes, and we should be better to
clarify the definition to avoid confusion on reading through the technical
doc. Shouldn't we use that term? I'm not sure. As long as we introduce a
new feature and name it, it is pretty much natural to consider the
marketing point of view for the name. IMHO, the reason 'real-time' is
referenced so many times in this industry (despite not being strictly
correct on the definition in the textbook) is probably because it is proved
to be very effective at marketing.
On Sat, May 31, 2025 at 4:40 AM Mark Hamstra <ma...@gmail.com> wrote:
> A soft real-time system still defines an interval or frame within which
> results should be available, and often provides explicit warning or
> error-handling mechanisms when frame rates are missed. I see nothing like
> that in the SPIP. Instead, the length of the underlying microbatches is
> specified in the Trigger, but result reporting is just as quickly as
> possible with no reporting interval or frame rate specified and nothing
> that I can see happening if results take longer than the user is guessing
> or expecting. That's a low-latency, "we'll do it as fast as we can, but no
> promises or guarantees" system, not real-time.
>
> On Thu, May 29, 2025 at 11:57 PM Jerry Peng <je...@gmail.com>
> wrote:
>
>> Mark,
>>
>> For real-time systems there is a concept of "soft" real-time and "hard"
>> real-time systems. These concepts exist in textbooks. Here is a document
>> by intel that explains it:
>>
>>
>> https://www.intel.com/content/www/us/en/learn/what-is-a-real-time-system.html
>>
>> "In a soft real-time system, computers or equipment will continue to
>> function after a missed deadline but may produce a lower-quality output.
>> For example, latency in online video games can impact player interactions,
>> but otherwise present no serious consequences."
>>
>> "Hard real-time systems have zero delay tolerance, and delayed signals
>> can result in total failure or present immediate danger to users. Flight
>> control systems and pacemakers are both examples where timeliness is not
>> only essential but the lack of it can result in a life-or-death situation."
>>
>> I don't think it is inaccurate or misleading to call this mode
>> real-time. It is soft real-time.
>>
>> On Thu, May 29, 2025 at 11:44 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> Clarifying what is meant by "real-time" and explicitly differentiating
>>> it from actual real-time computing should be a bare minimum. I still don't
>>> like the use of marketing-speak "real-time" that isn't really real-time in
>>> engineering documents or API namespaces.
>>>
>>> On Thu, May 29, 2025 at 10:43 PM Jerry Peng <je...@gmail.com>
>>> wrote:
>>>
>>>> Mark,
>>>>
>>>> I thought we are simply discussing the naming of the mode? Like I
>>>> mentioned, if you think simply calling this mode "real-time" mode may cause
>>>> confusion because "real-time" can mean other things in other fields, I can
>>>> clarify what we mean by "real-time" explicitly in the SPIP document and any
>>>> future documentation. That is not a problem and thank you for your feedback.
>>>>
>>>> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Referencing other misuse of "real-time" is not persuasive. A SPIP is
>>>>> an engineering document, not a marketing document. Technical clarity and
>>>>> accuracy should be non-negotiable.
>>>>>
>>>>>
>>>>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <
>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>
>>>>>> Mark,
>>>>>>
>>>>>> As an example of my point if you go the the Apache Storm (another
>>>>>> stream processing engine) website:
>>>>>>
>>>>>> https://storm.apache.org/
>>>>>>
>>>>>> It describes Storm as:
>>>>>>
>>>>>> "Apache Storm is a free and open source distributed *realtime*
>>>>>> computation system"
>>>>>>
>>>>>> If you can to apache Flink:
>>>>>>
>>>>>>
>>>>>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>>>>>
>>>>>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>>>>>
>>>>>> Thus, what the term "rea-time" implies in this should not be
>>>>>> confusing for folks in this area.
>>>>>>
>>>>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <
>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>
>>>>>>> Mich,
>>>>>>>
>>>>>>> If I understood your last email correctly, I think you also wanted
>>>>>>> to have a discussion about naming? Why are we calling this new execution
>>>>>>> mode described in the SPIP "Real-time Mode"? Here are my two cents.
>>>>>>> Firstly, "continuous mode" is taken and we want another name to describe an
>>>>>>> execution mode that provides ultra low latency processing. We could have
>>>>>>> called it "low latency mode", though I don't really like that naming since
>>>>>>> it implies the other execution modes are not low latency which I don't
>>>>>>> believe is true. This new proposed mode can simply deliver even lower
>>>>>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>>>>>> are talking about "soft" real-time here. I think when we are talking about
>>>>>>> distributed stream processing systems in the space of big data analytics,
>>>>>>> it is reasonable to assume anything described in this space as "real-time"
>>>>>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>>>>>> can provide clear documentation on what "real-time" in real-time mode means
>>>>>>> and what it guarantees. Just my thoughts. I would love to hear other
>>>>>>> perspectives.
>>>>>>>
>>>>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>
>>>>>>>> I think from what I have seen there are a good number of +1
>>>>>>>> responses as opposed to quantitative discussions (based on my observations
>>>>>>>> only). Given the objectives of the thread, we ought to focus on what is
>>>>>>>> meant by real time compared to continuous modes.To be fair, it is a
>>>>>>>> common point of confusion, and the terms are often used interchangeably in
>>>>>>>> general conversation, but in technical contexts, especially with streaming
>>>>>>>> data platforms, they have specific and important differences.
>>>>>>>>
>>>>>>>> "Continuous Mode" refers to a processing strategy that aims for
>>>>>>>> true, uninterrupted, sub-millisecond latency processing. Chiefly
>>>>>>>>
>>>>>>>> - Event-at-a-Time (or very small batch groups): The system
>>>>>>>> processes individual events or extremely small groups of events ->
>>>>>>>> microbatches as they flow through the pipeline.
>>>>>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>>>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>>>>>> even below
>>>>>>>> - Most business use cases (say financial markets) can live with
>>>>>>>> this as they do not rely on rdges
>>>>>>>>
>>>>>>>> Now what is meant by "Real-time Mode"
>>>>>>>>
>>>>>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>>>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>>>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>>>>>> that achieves real-time characteristics but might do so differently or more
>>>>>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>>>>>> mention, in real time application , there is nothing as an answer which is
>>>>>>>> supposed to be late and correct. The timeliness is part of the application.
>>>>>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>>>>>> call the "Late and Correct is Useless" Principle
>>>>>>>>
>>>>>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>>>>>> delivers low-latency processing with high reliability and ease of use,
>>>>>>>> leveraging established, battle-tested components.I invite the audience to
>>>>>>>> have a discussion on this.
>>>>>>>>
>>>>>>>> HTH
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>> GDPR
>>>>>>>>
>>>>>>>> view my Linkedin profile
>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>>>>>> > +1
>>>>>>>>> >
>>>>>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>>>>>> >
>>>>>>>>> > > +1.
>>>>>>>>> > >
>>>>>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com>
>>>>>>>>> wrote:
>>>>>>>>> > >
>>>>>>>>> > >> +1
>>>>>>>>> > >> Sent from my iPhone
>>>>>>>>> > >>
>>>>>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >> +1 Nice feature
>>>>>>>>> > >>
>>>>>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>>>>>> xyliyuanjian@gmail.com>
>>>>>>>>> > >> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >>> +1
>>>>>>>>> > >>>
>>>>>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>>>>>> > >>>
>>>>>>>>> > >>>> +1, LGTM.
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> Kent
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>>>>> > >>>>
>>>>>>>>> > >>>>> +1. Super excited by this initiative!
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>>>>>> ybliang8@gmail.com>
>>>>>>>>> > >>>>> wrote:
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>>> +1
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>>>>>> huaxin.gao11@gmail.com>
>>>>>>>>> > >>>>>> wrote:
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>>> +1
>>>>>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we
>>>>>>>>> can
>>>>>>>>> > >>>>>>> eliminate the need for separate streaming engines,
>>>>>>>>> reducing system
>>>>>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>>>>>> direction!
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>>> Hi,
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> My point about "in real time application or data, there
>>>>>>>>> is nothing
>>>>>>>>> > >>>>>>>> as an answer which is supposed to be late and correct.
>>>>>>>>> The timeliness is
>>>>>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>>>>>> slowly it becomes
>>>>>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we
>>>>>>>>> need this
>>>>>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of
>>>>>>>>> the answer
>>>>>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>>>>>> streaming
>>>>>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>>>>>> delivers results that
>>>>>>>>> > >>>>>>>> are technically "correct" but arrive too late to be
>>>>>>>>> truly useful for
>>>>>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes
>>>>>>>>> them "useless or
>>>>>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>>>>>> *high-frequency
>>>>>>>>> > >>>>>>>> trading,* market data or trade execution commands must
>>>>>>>>> be
>>>>>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>>>>>> mean missed
>>>>>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>>>>>> "correct" price
>>>>>>>>> > >>>>>>>> update useless if it's not instantaneous. able for
>>>>>>>>> these demanding
>>>>>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply
>>>>>>>>> not good enough. As
>>>>>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>>>>>> treated as such not
>>>>>>>>> > >>>>>>>> as a comment.in SPIP
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>>> Analysis |
>>>>>>>>> > >>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>>>>>> denny.g.lee@gmail.com>
>>>>>>>>> > >>>>>>>> wrote:
>>>>>>>>> > >>>>>>>>
>>>>>>>>> > >>>>>>>>> Hey Mich,
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does
>>>>>>>>> your
>>>>>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps
>>>>>>>>> add comments directly
>>>>>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet
>>>>>>>>> below is a direct copy
>>>>>>>>> > >>>>>>>>> from the SPIP itself.
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>>> > >>>>>>>>> Denny
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>
>>>>>>>>> > >>>>>>>>>> just to add
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>>>>>> definition of
>>>>>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>>>>>> application or
>>>>>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed
>>>>>>>>> to be late and
>>>>>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if
>>>>>>>>> I get the right
>>>>>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>>> Analysis |
>>>>>>>>> > >>>>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>>>>>> micro-batching.If you
>>>>>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction
>>>>>>>>> must be balanced against
>>>>>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>>>>>> prevent back pressure
>>>>>>>>> > >>>>>>>>>>> and instability. In the case of Continuous
>>>>>>>>> Processing mode, a
>>>>>>>>> > >>>>>>>>>>> specific continuous trigger with a desired
>>>>>>>>> checkpoint interval quote
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> "
>>>>>>>>> > >>>>>>>>>>> df.writeStream
>>>>>>>>> > >>>>>>>>>>> .format("...")
>>>>>>>>> > >>>>>>>>>>> .option("...")
>>>>>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) //
>>>>>>>>> new trigger
>>>>>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>>>>>> > >>>>>>>>>>> .start()
>>>>>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should
>>>>>>>>> run in the
>>>>>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time
>>>>>>>>> interval can also be
>>>>>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long
>>>>>>>>> each micro-batch should
>>>>>>>>> > >>>>>>>>>>> run for.
>>>>>>>>> > >>>>>>>>>>> "
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that
>>>>>>>>> simple
>>>>>>>>> > >>>>>>>>>>> HTH
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime |
>>>>>>>>> Forensic Analysis |
>>>>>>>>> > >>>>>>>>>>> GDPR
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>>>>>> > >>>>>>>>>>> <
>>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>> > >>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> Hi all,
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP
>>>>>>>>> titled
>>>>>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured
>>>>>>>>> Streaming” that I've been
>>>>>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao
>>>>>>>>> Sun, Jungtaek Lim, and
>>>>>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>>>>>> [Doc
>>>>>>>>> > >>>>>>>>>>>> <
>>>>>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>>>>>> >
>>>>>>>>> > >>>>>>>>>>>> ].
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>>>>>> “Real-time Mode”
>>>>>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly
>>>>>>>>> lowers end-to-end latency
>>>>>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility.
>>>>>>>>> Our goal is
>>>>>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs
>>>>>>>>> that need results almost
>>>>>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want
>>>>>>>>> to achieve this without
>>>>>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>>>>>> users already use – so
>>>>>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>>>>>> ultra-low-latency mode by
>>>>>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>>>>>> real-time
>>>>>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>>>>>> personalization) that
>>>>>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>>>>>> Spark’s current streaming
>>>>>>>>> > >>>>>>>>>>>> engine.
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>>>>>>>
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> --
>>>>>>>>> > >>>>>> Best,
>>>>>>>>> > >>>>>> Yanbo
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>
>>>>>>>>> > >>
>>>>>>>>> > >> --
>>>>>>>>> > >> John Zhuge
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>
>>>>>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mark Hamstra <ma...@gmail.com>.
A soft real-time system still defines an interval or frame within which
results should be available, and often provides explicit warning or
error-handling mechanisms when frame rates are missed. I see nothing like
that in the SPIP. Instead, the length of the underlying microbatches is
specified in the Trigger, but result reporting is just as quickly as
possible with no reporting interval or frame rate specified and nothing
that I can see happening if results take longer than the user is guessing
or expecting. That's a low-latency, "we'll do it as fast as we can, but no
promises or guarantees" system, not real-time.
On Thu, May 29, 2025 at 11:57 PM Jerry Peng <je...@gmail.com>
wrote:
> Mark,
>
> For real-time systems there is a concept of "soft" real-time and "hard"
> real-time systems. These concepts exist in textbooks. Here is a document
> by intel that explains it:
>
>
> https://www.intel.com/content/www/us/en/learn/what-is-a-real-time-system.html
>
> "In a soft real-time system, computers or equipment will continue to
> function after a missed deadline but may produce a lower-quality output.
> For example, latency in online video games can impact player interactions,
> but otherwise present no serious consequences."
>
> "Hard real-time systems have zero delay tolerance, and delayed signals can
> result in total failure or present immediate danger to users. Flight
> control systems and pacemakers are both examples where timeliness is not
> only essential but the lack of it can result in a life-or-death situation."
>
> I don't think it is inaccurate or misleading to call this mode real-time.
> It is soft real-time.
>
> On Thu, May 29, 2025 at 11:44 PM Mark Hamstra <ma...@gmail.com>
> wrote:
>
>> Clarifying what is meant by "real-time" and explicitly differentiating it
>> from actual real-time computing should be a bare minimum. I still don't
>> like the use of marketing-speak "real-time" that isn't really real-time in
>> engineering documents or API namespaces.
>>
>> On Thu, May 29, 2025 at 10:43 PM Jerry Peng <je...@gmail.com>
>> wrote:
>>
>>> Mark,
>>>
>>> I thought we are simply discussing the naming of the mode? Like I
>>> mentioned, if you think simply calling this mode "real-time" mode may cause
>>> confusion because "real-time" can mean other things in other fields, I can
>>> clarify what we mean by "real-time" explicitly in the SPIP document and any
>>> future documentation. That is not a problem and thank you for your feedback.
>>>
>>> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com>
>>> wrote:
>>>
>>>> Referencing other misuse of "real-time" is not persuasive. A SPIP is an
>>>> engineering document, not a marketing document. Technical clarity and
>>>> accuracy should be non-negotiable.
>>>>
>>>>
>>>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <
>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>
>>>>> Mark,
>>>>>
>>>>> As an example of my point if you go the the Apache Storm (another
>>>>> stream processing engine) website:
>>>>>
>>>>> https://storm.apache.org/
>>>>>
>>>>> It describes Storm as:
>>>>>
>>>>> "Apache Storm is a free and open source distributed *realtime*
>>>>> computation system"
>>>>>
>>>>> If you can to apache Flink:
>>>>>
>>>>>
>>>>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>>>>
>>>>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>>>>
>>>>> Thus, what the term "rea-time" implies in this should not be confusing
>>>>> for folks in this area.
>>>>>
>>>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <
>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>
>>>>>> Mich,
>>>>>>
>>>>>> If I understood your last email correctly, I think you also wanted to
>>>>>> have a discussion about naming? Why are we calling this new execution mode
>>>>>> described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
>>>>>> "continuous mode" is taken and we want another name to describe an
>>>>>> execution mode that provides ultra low latency processing. We could have
>>>>>> called it "low latency mode", though I don't really like that naming since
>>>>>> it implies the other execution modes are not low latency which I don't
>>>>>> believe is true. This new proposed mode can simply deliver even lower
>>>>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>>>>> are talking about "soft" real-time here. I think when we are talking about
>>>>>> distributed stream processing systems in the space of big data analytics,
>>>>>> it is reasonable to assume anything described in this space as "real-time"
>>>>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>>>>> can provide clear documentation on what "real-time" in real-time mode means
>>>>>> and what it guarantees. Just my thoughts. I would love to hear other
>>>>>> perspectives.
>>>>>>
>>>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> I think from what I have seen there are a good number of +1
>>>>>>> responses as opposed to quantitative discussions (based on my observations
>>>>>>> only). Given the objectives of the thread, we ought to focus on what is
>>>>>>> meant by real time compared to continuous modes.To be fair, it is a
>>>>>>> common point of confusion, and the terms are often used interchangeably in
>>>>>>> general conversation, but in technical contexts, especially with streaming
>>>>>>> data platforms, they have specific and important differences.
>>>>>>>
>>>>>>> "Continuous Mode" refers to a processing strategy that aims for
>>>>>>> true, uninterrupted, sub-millisecond latency processing. Chiefly
>>>>>>>
>>>>>>> - Event-at-a-Time (or very small batch groups): The system
>>>>>>> processes individual events or extremely small groups of events ->
>>>>>>> microbatches as they flow through the pipeline.
>>>>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>>>>> even below
>>>>>>> - Most business use cases (say financial markets) can live with
>>>>>>> this as they do not rely on rdges
>>>>>>>
>>>>>>> Now what is meant by "Real-time Mode"
>>>>>>>
>>>>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>>>>> that achieves real-time characteristics but might do so differently or more
>>>>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>>>>> mention, in real time application , there is nothing as an answer which is
>>>>>>> supposed to be late and correct. The timeliness is part of the application.
>>>>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>>>>> call the "Late and Correct is Useless" Principle
>>>>>>>
>>>>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>>>>> delivers low-latency processing with high reliability and ease of use,
>>>>>>> leveraging established, battle-tested components.I invite the audience to
>>>>>>> have a discussion on this.
>>>>>>>
>>>>>>> HTH
>>>>>>>
>>>>>>> Dr Mich Talebzadeh,
>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>>>>> > +1
>>>>>>>> >
>>>>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>>>>> >
>>>>>>>> > > +1.
>>>>>>>> > >
>>>>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com>
>>>>>>>> wrote:
>>>>>>>> > >
>>>>>>>> > >> +1
>>>>>>>> > >> Sent from my iPhone
>>>>>>>> > >>
>>>>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>>>>> wrote:
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >> +1 Nice feature
>>>>>>>> > >>
>>>>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>>>>> xyliyuanjian@gmail.com>
>>>>>>>> > >> wrote:
>>>>>>>> > >>
>>>>>>>> > >>> +1
>>>>>>>> > >>>
>>>>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>>>>> > >>>
>>>>>>>> > >>>> +1, LGTM.
>>>>>>>> > >>>>
>>>>>>>> > >>>> Kent
>>>>>>>> > >>>>
>>>>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>>>> > >>>>
>>>>>>>> > >>>>> +1. Super excited by this initiative!
>>>>>>>> > >>>>>
>>>>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>>>>> ybliang8@gmail.com>
>>>>>>>> > >>>>> wrote:
>>>>>>>> > >>>>>
>>>>>>>> > >>>>>> +1
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>>>>> huaxin.gao11@gmail.com>
>>>>>>>> > >>>>>> wrote:
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>>> +1
>>>>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we
>>>>>>>> can
>>>>>>>> > >>>>>>> eliminate the need for separate streaming engines,
>>>>>>>> reducing system
>>>>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>>>>> direction!
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>>> Hi,
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> My point about "in real time application or data, there
>>>>>>>> is nothing
>>>>>>>> > >>>>>>>> as an answer which is supposed to be late and correct.
>>>>>>>> The timeliness is
>>>>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>>>>> slowly it becomes
>>>>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we
>>>>>>>> need this
>>>>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of
>>>>>>>> the answer
>>>>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>>>>> streaming
>>>>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>>>>> delivers results that
>>>>>>>> > >>>>>>>> are technically "correct" but arrive too late to be
>>>>>>>> truly useful for
>>>>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes
>>>>>>>> them "useless or
>>>>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>>>>> *high-frequency
>>>>>>>> > >>>>>>>> trading,* market data or trade execution commands must be
>>>>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>>>>> mean missed
>>>>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>>>>> "correct" price
>>>>>>>> > >>>>>>>> update useless if it's not instantaneous. able for these
>>>>>>>> demanding
>>>>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply
>>>>>>>> not good enough. As
>>>>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>>>>> treated as such not
>>>>>>>> > >>>>>>>> as a comment.in SPIP
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>> Analysis |
>>>>>>>> > >>>>>>>> GDPR
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> view my Linkedin profile
>>>>>>>> > >>>>>>>> <
>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>>>>> denny.g.lee@gmail.com>
>>>>>>>> > >>>>>>>> wrote:
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>> Hey Mich,
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does
>>>>>>>> your
>>>>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>>>>>>>> comments directly
>>>>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet
>>>>>>>> below is a direct copy
>>>>>>>> > >>>>>>>>> from the SPIP itself.
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> Thanks,
>>>>>>>> > >>>>>>>>> Denny
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>> just to add
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>>>>> definition of
>>>>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>>>>> application or
>>>>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed
>>>>>>>> to be late and
>>>>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if
>>>>>>>> I get the right
>>>>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>> Analysis |
>>>>>>>> > >>>>>>>>>> GDPR
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>>>>> > >>>>>>>>>> <
>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>>>>> micro-batching.If you
>>>>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction
>>>>>>>> must be balanced against
>>>>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>>>>> prevent back pressure
>>>>>>>> > >>>>>>>>>>> and instability. In the case of Continuous Processing
>>>>>>>> mode, a
>>>>>>>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>>>>>>>> interval quote
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> "
>>>>>>>> > >>>>>>>>>>> df.writeStream
>>>>>>>> > >>>>>>>>>>> .format("...")
>>>>>>>> > >>>>>>>>>>> .option("...")
>>>>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) //
>>>>>>>> new trigger
>>>>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>>>>> > >>>>>>>>>>> .start()
>>>>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should
>>>>>>>> run in the
>>>>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time
>>>>>>>> interval can also be
>>>>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long
>>>>>>>> each micro-batch should
>>>>>>>> > >>>>>>>>>>> run for.
>>>>>>>> > >>>>>>>>>>> "
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that
>>>>>>>> simple
>>>>>>>> > >>>>>>>>>>> HTH
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>>> Analysis |
>>>>>>>> > >>>>>>>>>>> GDPR
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>>>>> > >>>>>>>>>>> <
>>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> Hi all,
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP
>>>>>>>> titled
>>>>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured
>>>>>>>> Streaming” that I've been
>>>>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>>>>>>>> Jungtaek Lim, and
>>>>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>>>>> [Doc
>>>>>>>> > >>>>>>>>>>>> <
>>>>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>>>>> >
>>>>>>>> > >>>>>>>>>>>> ].
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>>>>> “Real-time Mode”
>>>>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly
>>>>>>>> lowers end-to-end latency
>>>>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility.
>>>>>>>> Our goal is
>>>>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs
>>>>>>>> that need results almost
>>>>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>>>>>>>> achieve this without
>>>>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>>>>> users already use – so
>>>>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>>>>> ultra-low-latency mode by
>>>>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>>>>> real-time
>>>>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>>>>> personalization) that
>>>>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>>>>> Spark’s current streaming
>>>>>>>> > >>>>>>>>>>>> engine.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> --
>>>>>>>> > >>>>>> Best,
>>>>>>>> > >>>>>> Yanbo
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>
>>>>>>>> > >>
>>>>>>>> > >> --
>>>>>>>> > >> John Zhuge
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jerry Peng <je...@gmail.com>.
Mark,
For real-time systems there is a concept of "soft" real-time and "hard"
real-time systems. These concepts exist in textbooks. Here is a document
by intel that explains it:
https://www.intel.com/content/www/us/en/learn/what-is-a-real-time-system.html
"In a soft real-time system, computers or equipment will continue to
function after a missed deadline but may produce a lower-quality output.
For example, latency in online video games can impact player interactions,
but otherwise present no serious consequences."
"Hard real-time systems have zero delay tolerance, and delayed signals can
result in total failure or present immediate danger to users. Flight
control systems and pacemakers are both examples where timeliness is not
only essential but the lack of it can result in a life-or-death situation."
I don't think it is inaccurate or misleading to call this mode real-time.
It is soft real-time.
On Thu, May 29, 2025 at 11:44 PM Mark Hamstra <ma...@gmail.com> wrote:
> Clarifying what is meant by "real-time" and explicitly differentiating it
> from actual real-time computing should be a bare minimum. I still don't
> like the use of marketing-speak "real-time" that isn't really real-time in
> engineering documents or API namespaces.
>
> On Thu, May 29, 2025 at 10:43 PM Jerry Peng <je...@gmail.com>
> wrote:
>
>> Mark,
>>
>> I thought we are simply discussing the naming of the mode? Like I
>> mentioned, if you think simply calling this mode "real-time" mode may cause
>> confusion because "real-time" can mean other things in other fields, I can
>> clarify what we mean by "real-time" explicitly in the SPIP document and any
>> future documentation. That is not a problem and thank you for your feedback.
>>
>> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> Referencing other misuse of "real-time" is not persuasive. A SPIP is an
>>> engineering document, not a marketing document. Technical clarity and
>>> accuracy should be non-negotiable.
>>>
>>>
>>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <je...@gmail.com>
>>> wrote:
>>>
>>>> Mark,
>>>>
>>>> As an example of my point if you go the the Apache Storm (another
>>>> stream processing engine) website:
>>>>
>>>> https://storm.apache.org/
>>>>
>>>> It describes Storm as:
>>>>
>>>> "Apache Storm is a free and open source distributed *realtime*
>>>> computation system"
>>>>
>>>> If you can to apache Flink:
>>>>
>>>>
>>>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>>>
>>>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>>>
>>>> Thus, what the term "rea-time" implies in this should not be confusing
>>>> for folks in this area.
>>>>
>>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <
>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>
>>>>> Mich,
>>>>>
>>>>> If I understood your last email correctly, I think you also wanted to
>>>>> have a discussion about naming? Why are we calling this new execution mode
>>>>> described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
>>>>> "continuous mode" is taken and we want another name to describe an
>>>>> execution mode that provides ultra low latency processing. We could have
>>>>> called it "low latency mode", though I don't really like that naming since
>>>>> it implies the other execution modes are not low latency which I don't
>>>>> believe is true. This new proposed mode can simply deliver even lower
>>>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>>>> are talking about "soft" real-time here. I think when we are talking about
>>>>> distributed stream processing systems in the space of big data analytics,
>>>>> it is reasonable to assume anything described in this space as "real-time"
>>>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>>>> can provide clear documentation on what "real-time" in real-time mode means
>>>>> and what it guarantees. Just my thoughts. I would love to hear other
>>>>> perspectives.
>>>>>
>>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> I think from what I have seen there are a good number of +1 responses
>>>>>> as opposed to quantitative discussions (based on my observations only).
>>>>>> Given the objectives of the thread, we ought to focus on what is meant by
>>>>>> real time compared to continuous modes.To be fair, it is a common point
>>>>>> of confusion, and the terms are often used interchangeably in general
>>>>>> conversation, but in technical contexts, especially with streaming data
>>>>>> platforms, they have specific and important differences.
>>>>>>
>>>>>> "Continuous Mode" refers to a processing strategy that aims for true,
>>>>>> uninterrupted, sub-millisecond latency processing. Chiefly
>>>>>>
>>>>>> - Event-at-a-Time (or very small batch groups): The system
>>>>>> processes individual events or extremely small groups of events ->
>>>>>> microbatches as they flow through the pipeline.
>>>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>>>> even below
>>>>>> - Most business use cases (say financial markets) can live with
>>>>>> this as they do not rely on rdges
>>>>>>
>>>>>> Now what is meant by "Real-time Mode"
>>>>>>
>>>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>>>> that achieves real-time characteristics but might do so differently or more
>>>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>>>> mention, in real time application , there is nothing as an answer which is
>>>>>> supposed to be late and correct. The timeliness is part of the application.
>>>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>>>> call the "Late and Correct is Useless" Principle
>>>>>>
>>>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>>>> delivers low-latency processing with high reliability and ease of use,
>>>>>> leveraging established, battle-tested components.I invite the audience to
>>>>>> have a discussion on this.
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>> Dr Mich Talebzadeh,
>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>>>> > +1
>>>>>>> >
>>>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>>>> >
>>>>>>> > > +1.
>>>>>>> > >
>>>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > >> +1
>>>>>>> > >> Sent from my iPhone
>>>>>>> > >>
>>>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>>>> wrote:
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> +1 Nice feature
>>>>>>> > >>
>>>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>>>> xyliyuanjian@gmail.com>
>>>>>>> > >> wrote:
>>>>>>> > >>
>>>>>>> > >>> +1
>>>>>>> > >>>
>>>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>>>> > >>>
>>>>>>> > >>>> +1, LGTM.
>>>>>>> > >>>>
>>>>>>> > >>>> Kent
>>>>>>> > >>>>
>>>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>>> > >>>>
>>>>>>> > >>>>> +1. Super excited by this initiative!
>>>>>>> > >>>>>
>>>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>>>> ybliang8@gmail.com>
>>>>>>> > >>>>> wrote:
>>>>>>> > >>>>>
>>>>>>> > >>>>>> +1
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>>>> huaxin.gao11@gmail.com>
>>>>>>> > >>>>>> wrote:
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>> +1
>>>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we
>>>>>>> can
>>>>>>> > >>>>>>> eliminate the need for separate streaming engines,
>>>>>>> reducing system
>>>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>>>> direction!
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>>> Hi,
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> My point about "in real time application or data, there
>>>>>>> is nothing
>>>>>>> > >>>>>>>> as an answer which is supposed to be late and correct.
>>>>>>> The timeliness is
>>>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>>>> slowly it becomes
>>>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we
>>>>>>> need this
>>>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of
>>>>>>> the answer
>>>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>>>> streaming
>>>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>>>> delivers results that
>>>>>>> > >>>>>>>> are technically "correct" but arrive too late to be truly
>>>>>>> useful for
>>>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
>>>>>>> "useless or
>>>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>>>> *high-frequency
>>>>>>> > >>>>>>>> trading,* market data or trade execution commands must be
>>>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>>>> mean missed
>>>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>>>> "correct" price
>>>>>>> > >>>>>>>> update useless if it's not instantaneous. able for these
>>>>>>> demanding
>>>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply
>>>>>>> not good enough. As
>>>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>>>> treated as such not
>>>>>>> > >>>>>>>> as a comment.in SPIP
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>> Analysis |
>>>>>>> > >>>>>>>> GDPR
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> view my Linkedin profile
>>>>>>> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/
>>>>>>> >
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>>>> denny.g.lee@gmail.com>
>>>>>>> > >>>>>>>> wrote:
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>>> Hey Mich,
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does your
>>>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>>>>>>> comments directly
>>>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet below
>>>>>>> is a direct copy
>>>>>>> > >>>>>>>>> from the SPIP itself.
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> Thanks,
>>>>>>> > >>>>>>>>> Denny
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>> just to add
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>>>> definition of
>>>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>>>> application or
>>>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed
>>>>>>> to be late and
>>>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if I
>>>>>>> get the right
>>>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>> Analysis |
>>>>>>> > >>>>>>>>>> GDPR
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>>>> > >>>>>>>>>> <
>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>>>> micro-batching.If you
>>>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction
>>>>>>> must be balanced against
>>>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>>>> prevent back pressure
>>>>>>> > >>>>>>>>>>> and instability. In the case of Continuous Processing
>>>>>>> mode, a
>>>>>>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>>>>>>> interval quote
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> "
>>>>>>> > >>>>>>>>>>> df.writeStream
>>>>>>> > >>>>>>>>>>> .format("...")
>>>>>>> > >>>>>>>>>>> .option("...")
>>>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
>>>>>>> trigger
>>>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>>>> > >>>>>>>>>>> .start()
>>>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should
>>>>>>> run in the
>>>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval
>>>>>>> can also be
>>>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long
>>>>>>> each micro-batch should
>>>>>>> > >>>>>>>>>>> run for.
>>>>>>> > >>>>>>>>>>> "
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>> > >>>>>>>>>>> HTH
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>>> Analysis |
>>>>>>> > >>>>>>>>>>> GDPR
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>>>> > >>>>>>>>>>> <
>>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> Hi all,
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP
>>>>>>> titled
>>>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming”
>>>>>>> that I've been
>>>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>>>>>>> Jungtaek Lim, and
>>>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>>>> [Doc
>>>>>>> > >>>>>>>>>>>> <
>>>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>>>> >
>>>>>>> > >>>>>>>>>>>> ].
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>>>> “Real-time Mode”
>>>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly
>>>>>>> lowers end-to-end latency
>>>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility.
>>>>>>> Our goal is
>>>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that
>>>>>>> need results almost
>>>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>>>>>>> achieve this without
>>>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>>>> users already use – so
>>>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>>>> ultra-low-latency mode by
>>>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>>>> real-time
>>>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>>>> personalization) that
>>>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>>>> Spark’s current streaming
>>>>>>> > >>>>>>>>>>>> engine.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> --
>>>>>>> > >>>>>> Best,
>>>>>>> > >>>>>> Yanbo
>>>>>>> > >>>>>>
>>>>>>> > >>>>>
>>>>>>> > >>
>>>>>>> > >> --
>>>>>>> > >> John Zhuge
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> >
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mark Hamstra <ma...@gmail.com>.
Clarifying what is meant by "real-time" and explicitly differentiating it
from actual real-time computing should be a bare minimum. I still don't
like the use of marketing-speak "real-time" that isn't really real-time in
engineering documents or API namespaces.
On Thu, May 29, 2025 at 10:43 PM Jerry Peng <je...@gmail.com>
wrote:
> Mark,
>
> I thought we are simply discussing the naming of the mode? Like I
> mentioned, if you think simply calling this mode "real-time" mode may cause
> confusion because "real-time" can mean other things in other fields, I can
> clarify what we mean by "real-time" explicitly in the SPIP document and any
> future documentation. That is not a problem and thank you for your feedback.
>
> On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com>
> wrote:
>
>> Referencing other misuse of "real-time" is not persuasive. A SPIP is an
>> engineering document, not a marketing document. Technical clarity and
>> accuracy should be non-negotiable.
>>
>>
>> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <je...@gmail.com>
>> wrote:
>>
>>> Mark,
>>>
>>> As an example of my point if you go the the Apache Storm (another stream
>>> processing engine) website:
>>>
>>> https://storm.apache.org/
>>>
>>> It describes Storm as:
>>>
>>> "Apache Storm is a free and open source distributed *realtime*
>>> computation system"
>>>
>>> If you can to apache Flink:
>>>
>>>
>>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>>
>>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>>
>>> Thus, what the term "rea-time" implies in this should not be confusing
>>> for folks in this area.
>>>
>>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <je...@gmail.com>
>>> wrote:
>>>
>>>> Mich,
>>>>
>>>> If I understood your last email correctly, I think you also wanted to
>>>> have a discussion about naming? Why are we calling this new execution mode
>>>> described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
>>>> "continuous mode" is taken and we want another name to describe an
>>>> execution mode that provides ultra low latency processing. We could have
>>>> called it "low latency mode", though I don't really like that naming since
>>>> it implies the other execution modes are not low latency which I don't
>>>> believe is true. This new proposed mode can simply deliver even lower
>>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>>> are talking about "soft" real-time here. I think when we are talking about
>>>> distributed stream processing systems in the space of big data analytics,
>>>> it is reasonable to assume anything described in this space as "real-time"
>>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>>> can provide clear documentation on what "real-time" in real-time mode means
>>>> and what it guarantees. Just my thoughts. I would love to hear other
>>>> perspectives.
>>>>
>>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> I think from what I have seen there are a good number of +1 responses
>>>>> as opposed to quantitative discussions (based on my observations only).
>>>>> Given the objectives of the thread, we ought to focus on what is meant by
>>>>> real time compared to continuous modes.To be fair, it is a common point
>>>>> of confusion, and the terms are often used interchangeably in general
>>>>> conversation, but in technical contexts, especially with streaming data
>>>>> platforms, they have specific and important differences.
>>>>>
>>>>> "Continuous Mode" refers to a processing strategy that aims for true,
>>>>> uninterrupted, sub-millisecond latency processing. Chiefly
>>>>>
>>>>> - Event-at-a-Time (or very small batch groups): The system
>>>>> processes individual events or extremely small groups of events ->
>>>>> microbatches as they flow through the pipeline.
>>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>>> even below
>>>>> - Most business use cases (say financial markets) can live with
>>>>> this as they do not rely on rdges
>>>>>
>>>>> Now what is meant by "Real-time Mode"
>>>>>
>>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>>> that achieves real-time characteristics but might do so differently or more
>>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>>> mention, in real time application , there is nothing as an answer which is
>>>>> supposed to be late and correct. The timeliness is part of the application.
>>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>>> call the "Late and Correct is Useless" Principle
>>>>>
>>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>>> delivers low-latency processing with high reliability and ease of use,
>>>>> leveraging established, battle-tested components.I invite the audience to
>>>>> have a discussion on this.
>>>>>
>>>>> HTH
>>>>>
>>>>> Dr Mich Talebzadeh,
>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>>> > +1
>>>>>> >
>>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>>> >
>>>>>> > > +1.
>>>>>> > >
>>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com>
>>>>>> wrote:
>>>>>> > >
>>>>>> > >> +1
>>>>>> > >> Sent from my iPhone
>>>>>> > >>
>>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>>> wrote:
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> +1 Nice feature
>>>>>> > >>
>>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>>> xyliyuanjian@gmail.com>
>>>>>> > >> wrote:
>>>>>> > >>
>>>>>> > >>> +1
>>>>>> > >>>
>>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>>> > >>>
>>>>>> > >>>> +1, LGTM.
>>>>>> > >>>>
>>>>>> > >>>> Kent
>>>>>> > >>>>
>>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>> > >>>>
>>>>>> > >>>>> +1. Super excited by this initiative!
>>>>>> > >>>>>
>>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>>> ybliang8@gmail.com>
>>>>>> > >>>>> wrote:
>>>>>> > >>>>>
>>>>>> > >>>>>> +1
>>>>>> > >>>>>>
>>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>>> huaxin.gao11@gmail.com>
>>>>>> > >>>>>> wrote:
>>>>>> > >>>>>>
>>>>>> > >>>>>>> +1
>>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>>>>> > >>>>>>> eliminate the need for separate streaming engines, reducing
>>>>>> system
>>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>>> direction!
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>> > >>>>>>>
>>>>>> > >>>>>>>> Hi,
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> My point about "in real time application or data, there is
>>>>>> nothing
>>>>>> > >>>>>>>> as an answer which is supposed to be late and correct. The
>>>>>> timeliness is
>>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>>> slowly it becomes
>>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need
>>>>>> this
>>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of
>>>>>> the answer
>>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>>> streaming
>>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>>> delivers results that
>>>>>> > >>>>>>>> are technically "correct" but arrive too late to be truly
>>>>>> useful for
>>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
>>>>>> "useless or
>>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>>> *high-frequency
>>>>>> > >>>>>>>> trading,* market data or trade execution commands must be
>>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>>> mean missed
>>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>>> "correct" price
>>>>>> > >>>>>>>> update useless if it's not instantaneous. able for these
>>>>>> demanding
>>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply not
>>>>>> good enough. As
>>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>>> treated as such not
>>>>>> > >>>>>>>> as a comment.in SPIP
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>> Analysis |
>>>>>> > >>>>>>>> GDPR
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> view my Linkedin profile
>>>>>> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>>> denny.g.lee@gmail.com>
>>>>>> > >>>>>>>> wrote:
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>>> Hey Mich,
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does your
>>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>>>>>> comments directly
>>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet below
>>>>>> is a direct copy
>>>>>> > >>>>>>>>> from the SPIP itself.
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> Thanks,
>>>>>> > >>>>>>>>> Denny
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>>> just to add
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>>> definition of
>>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>>> application or
>>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed to
>>>>>> be late and
>>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if I
>>>>>> get the right
>>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>> Analysis |
>>>>>> > >>>>>>>>>> GDPR
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>>> > >>>>>>>>>> <
>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>>> micro-batching.If you
>>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must
>>>>>> be balanced against
>>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>>> prevent back pressure
>>>>>> > >>>>>>>>>>> and instability. In the case of Continuous Processing
>>>>>> mode, a
>>>>>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>>>>>> interval quote
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> "
>>>>>> > >>>>>>>>>>> df.writeStream
>>>>>> > >>>>>>>>>>> .format("...")
>>>>>> > >>>>>>>>>>> .option("...")
>>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
>>>>>> trigger
>>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>>> > >>>>>>>>>>> .start()
>>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run
>>>>>> in the
>>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval
>>>>>> can also be
>>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long
>>>>>> each micro-batch should
>>>>>> > >>>>>>>>>>> run for.
>>>>>> > >>>>>>>>>>> "
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>> > >>>>>>>>>>> HTH
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>>> Analysis |
>>>>>> > >>>>>>>>>>> GDPR
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>>> > >>>>>>>>>>> <
>>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> Hi all,
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming”
>>>>>> that I've been
>>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>>>>>> Jungtaek Lim, and
>>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>>> [Doc
>>>>>> > >>>>>>>>>>>> <
>>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>>> >
>>>>>> > >>>>>>>>>>>> ].
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>>> “Real-time Mode”
>>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly
>>>>>> lowers end-to-end latency
>>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our
>>>>>> goal is
>>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that
>>>>>> need results almost
>>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>>>>>> achieve this without
>>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>>> users already use – so
>>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>>> ultra-low-latency mode by
>>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>>> real-time
>>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>>> personalization) that
>>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>>> Spark’s current streaming
>>>>>> > >>>>>>>>>>>> engine.
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>
>>>>>> > >>>>>> --
>>>>>> > >>>>>> Best,
>>>>>> > >>>>>> Yanbo
>>>>>> > >>>>>>
>>>>>> > >>>>>
>>>>>> > >>
>>>>>> > >> --
>>>>>> > >> John Zhuge
>>>>>> > >>
>>>>>> > >>
>>>>>> >
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jerry Peng <je...@gmail.com>.
Mark,
I thought we are simply discussing the naming of the mode? Like I
mentioned, if you think simply calling this mode "real-time" mode may cause
confusion because "real-time" can mean other things in other fields, I can
clarify what we mean by "real-time" explicitly in the SPIP document and any
future documentation. That is not a problem and thank you for your feedback.
On Thu, May 29, 2025 at 10:37 PM Mark Hamstra <ma...@gmail.com> wrote:
> Referencing other misuse of "real-time" is not persuasive. A SPIP is an
> engineering document, not a marketing document. Technical clarity and
> accuracy should be non-negotiable.
>
>
> On Thu, May 29, 2025 at 10:27 PM Jerry Peng <je...@gmail.com>
> wrote:
>
>> Mark,
>>
>> As an example of my point if you go the the Apache Storm (another stream
>> processing engine) website:
>>
>> https://storm.apache.org/
>>
>> It describes Storm as:
>>
>> "Apache Storm is a free and open source distributed *realtime*
>> computation system"
>>
>> If you can to apache Flink:
>>
>>
>> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>>
>> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>>
>> Thus, what the term "rea-time" implies in this should not be confusing
>> for folks in this area.
>>
>> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <je...@gmail.com>
>> wrote:
>>
>>> Mich,
>>>
>>> If I understood your last email correctly, I think you also wanted to
>>> have a discussion about naming? Why are we calling this new execution mode
>>> described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
>>> "continuous mode" is taken and we want another name to describe an
>>> execution mode that provides ultra low latency processing. We could have
>>> called it "low latency mode", though I don't really like that naming since
>>> it implies the other execution modes are not low latency which I don't
>>> believe is true. This new proposed mode can simply deliver even lower
>>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>>> are talking about "soft" real-time here. I think when we are talking about
>>> distributed stream processing systems in the space of big data analytics,
>>> it is reasonable to assume anything described in this space as "real-time"
>>> implies "soft" real-time. Though if this is confusing or misleading, we
>>> can provide clear documentation on what "real-time" in real-time mode means
>>> and what it guarantees. Just my thoughts. I would love to hear other
>>> perspectives.
>>>
>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> I think from what I have seen there are a good number of +1 responses
>>>> as opposed to quantitative discussions (based on my observations only).
>>>> Given the objectives of the thread, we ought to focus on what is meant by
>>>> real time compared to continuous modes.To be fair, it is a common point
>>>> of confusion, and the terms are often used interchangeably in general
>>>> conversation, but in technical contexts, especially with streaming data
>>>> platforms, they have specific and important differences.
>>>>
>>>> "Continuous Mode" refers to a processing strategy that aims for true,
>>>> uninterrupted, sub-millisecond latency processing. Chiefly
>>>>
>>>> - Event-at-a-Time (or very small batch groups): The system
>>>> processes individual events or extremely small groups of events ->
>>>> microbatches as they flow through the pipeline.
>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>> even below
>>>> - Most business use cases (say financial markets) can live with
>>>> this as they do not rely on rdges
>>>>
>>>> Now what is meant by "Real-time Mode"
>>>>
>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>> that achieves real-time characteristics but might do so differently or more
>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>> mention, in real time application , there is nothing as an answer which is
>>>> supposed to be late and correct. The timeliness is part of the application.
>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>> call the "Late and Correct is Useless" Principle
>>>>
>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>> delivers low-latency processing with high reliability and ease of use,
>>>> leveraging established, battle-tested components.I invite the audience to
>>>> have a discussion on this.
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh,
>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>> > +1
>>>>> >
>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>> >
>>>>> > > +1.
>>>>> > >
>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
>>>>> > >
>>>>> > >> +1
>>>>> > >> Sent from my iPhone
>>>>> > >>
>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>> wrote:
>>>>> > >>
>>>>> > >>
>>>>> > >> +1 Nice feature
>>>>> > >>
>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>> xyliyuanjian@gmail.com>
>>>>> > >> wrote:
>>>>> > >>
>>>>> > >>> +1
>>>>> > >>>
>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>> > >>>
>>>>> > >>>> +1, LGTM.
>>>>> > >>>>
>>>>> > >>>> Kent
>>>>> > >>>>
>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>> > >>>>
>>>>> > >>>>> +1. Super excited by this initiative!
>>>>> > >>>>>
>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>> ybliang8@gmail.com>
>>>>> > >>>>> wrote:
>>>>> > >>>>>
>>>>> > >>>>>> +1
>>>>> > >>>>>>
>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>> huaxin.gao11@gmail.com>
>>>>> > >>>>>> wrote:
>>>>> > >>>>>>
>>>>> > >>>>>>> +1
>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>>>> > >>>>>>> eliminate the need for separate streaming engines, reducing
>>>>> system
>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>> direction!
>>>>> > >>>>>>>
>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>> > >>>>>>>
>>>>> > >>>>>>>> Hi,
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> My point about "in real time application or data, there is
>>>>> nothing
>>>>> > >>>>>>>> as an answer which is supposed to be late and correct. The
>>>>> timeliness is
>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>> slowly it becomes
>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need
>>>>> this
>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of the
>>>>> answer
>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>> streaming
>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>> delivers results that
>>>>> > >>>>>>>> are technically "correct" but arrive too late to be truly
>>>>> useful for
>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
>>>>> "useless or
>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>> *high-frequency
>>>>> > >>>>>>>> trading,* market data or trade execution commands must be
>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>> mean missed
>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>> "correct" price
>>>>> > >>>>>>>> update useless if it's not instantaneous. able for these
>>>>> demanding
>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply not
>>>>> good enough. As
>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>> treated as such not
>>>>> > >>>>>>>> as a comment.in SPIP
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>> Analysis |
>>>>> > >>>>>>>> GDPR
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> view my Linkedin profile
>>>>> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>> denny.g.lee@gmail.com>
>>>>> > >>>>>>>> wrote:
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>> Hey Mich,
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does your
>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>>>>> comments directly
>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet below
>>>>> is a direct copy
>>>>> > >>>>>>>>> from the SPIP itself.
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Thanks,
>>>>> > >>>>>>>>> Denny
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>> just to add
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>> definition of
>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>> application or
>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed to
>>>>> be late and
>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if I
>>>>> get the right
>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>> Analysis |
>>>>> > >>>>>>>>>> GDPR
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>> > >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/
>>>>> >
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>> micro-batching.If you
>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must
>>>>> be balanced against
>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>> prevent back pressure
>>>>> > >>>>>>>>>>> and instability. In the case of Continuous Processing
>>>>> mode, a
>>>>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>>>>> interval quote
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> "
>>>>> > >>>>>>>>>>> df.writeStream
>>>>> > >>>>>>>>>>> .format("...")
>>>>> > >>>>>>>>>>> .option("...")
>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
>>>>> trigger
>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>> > >>>>>>>>>>> .start()
>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run
>>>>> in the
>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval
>>>>> can also be
>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each
>>>>> micro-batch should
>>>>> > >>>>>>>>>>> run for.
>>>>> > >>>>>>>>>>> "
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>> > >>>>>>>>>>> HTH
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>> Analysis |
>>>>> > >>>>>>>>>>> GDPR
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>> > >>>>>>>>>>> <
>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>> Hi all,
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming”
>>>>> that I've been
>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>>>>> Jungtaek Lim, and
>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>> [Doc
>>>>> > >>>>>>>>>>>> <
>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>> >
>>>>> > >>>>>>>>>>>> ].
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>> “Real-time Mode”
>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers
>>>>> end-to-end latency
>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our
>>>>> goal is
>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that
>>>>> need results almost
>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>>>>> achieve this without
>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>> users already use – so
>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>> ultra-low-latency mode by
>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>> real-time
>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>> personalization) that
>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>> Spark’s current streaming
>>>>> > >>>>>>>>>>>> engine.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>
>>>>> > >>>>>> --
>>>>> > >>>>>> Best,
>>>>> > >>>>>> Yanbo
>>>>> > >>>>>>
>>>>> > >>>>>
>>>>> > >>
>>>>> > >> --
>>>>> > >> John Zhuge
>>>>> > >>
>>>>> > >>
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mark Hamstra <ma...@gmail.com>.
Referencing other misuse of "real-time" is not persuasive. A SPIP is an
engineering document, not a marketing document. Technical clarity and
accuracy should be non-negotiable.
On Thu, May 29, 2025 at 10:27 PM Jerry Peng <je...@gmail.com>
wrote:
> Mark,
>
> As an example of my point if you go the the Apache Storm (another stream
> processing engine) website:
>
> https://storm.apache.org/
>
> It describes Storm as:
>
> "Apache Storm is a free and open source distributed *realtime*
> computation system"
>
> If you can to apache Flink:
>
>
> https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
>
> "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
>
> Thus, what the term "rea-time" implies in this should not be confusing for
> folks in this area.
>
> On Thu, May 29, 2025 at 10:22 PM Jerry Peng <je...@gmail.com>
> wrote:
>
>> Mich,
>>
>> If I understood your last email correctly, I think you also wanted to
>> have a discussion about naming? Why are we calling this new execution mode
>> described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
>> "continuous mode" is taken and we want another name to describe an
>> execution mode that provides ultra low latency processing. We could have
>> called it "low latency mode", though I don't really like that naming since
>> it implies the other execution modes are not low latency which I don't
>> believe is true. This new proposed mode can simply deliver even lower
>> latency. Thus, we came up with the name "Real-time Mode". Of course, we
>> are talking about "soft" real-time here. I think when we are talking about
>> distributed stream processing systems in the space of big data analytics,
>> it is reasonable to assume anything described in this space as "real-time"
>> implies "soft" real-time. Though if this is confusing or misleading, we
>> can provide clear documentation on what "real-time" in real-time mode means
>> and what it guarantees. Just my thoughts. I would love to hear other
>> perspectives.
>>
>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> I think from what I have seen there are a good number of +1 responses as
>>> opposed to quantitative discussions (based on my observations only). Given
>>> the objectives of the thread, we ought to focus on what is meant by real
>>> time compared to continuous modes.To be fair, it is a common point of
>>> confusion, and the terms are often used interchangeably in general
>>> conversation, but in technical contexts, especially with streaming data
>>> platforms, they have specific and important differences.
>>>
>>> "Continuous Mode" refers to a processing strategy that aims for true,
>>> uninterrupted, sub-millisecond latency processing. Chiefly
>>>
>>> - Event-at-a-Time (or very small batch groups): The system
>>> processes individual events or extremely small groups of events ->
>>> microbatches as they flow through the pipeline.
>>> - Minimal Latency: The primary goal is to achieve the absolute
>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>> even below
>>> - Most business use cases (say financial markets) can live with this
>>> as they do not rely on rdges
>>>
>>> Now what is meant by "Real-time Mode"
>>>
>>> This is where the nuance comes in. "Real-time" is a broader and
>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>> that achieves real-time characteristics but might do so differently or more
>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>> mention, in real time application , there is nothing as an answer which is
>>> supposed to be late and correct. The timeliness is part of the application.
>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>> call the "Late and Correct is Useless" Principle
>>>
>>> In summary, "Real-time Mode" seems to describe an approach that delivers
>>> low-latency processing with high reliability and ease of use, leveraging
>>> established, battle-tested components.I invite the audience to have a
>>> discussion on this.
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>>>
>>>> +1
>>>>
>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>> > +1
>>>> >
>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>> >
>>>> > > +1.
>>>> > >
>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
>>>> > >
>>>> > >> +1
>>>> > >> Sent from my iPhone
>>>> > >>
>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>> wrote:
>>>> > >>
>>>> > >>
>>>> > >> +1 Nice feature
>>>> > >>
>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>> xyliyuanjian@gmail.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >>> +1
>>>> > >>>
>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>> > >>>
>>>> > >>>> +1, LGTM.
>>>> > >>>>
>>>> > >>>> Kent
>>>> > >>>>
>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>> > >>>>
>>>> > >>>>> +1. Super excited by this initiative!
>>>> > >>>>>
>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <ybliang8@gmail.com
>>>> >
>>>> > >>>>> wrote:
>>>> > >>>>>
>>>> > >>>>>> +1
>>>> > >>>>>>
>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>> huaxin.gao11@gmail.com>
>>>> > >>>>>> wrote:
>>>> > >>>>>>
>>>> > >>>>>>> +1
>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>>> > >>>>>>> eliminate the need for separate streaming engines, reducing
>>>> system
>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>> direction!
>>>> > >>>>>>>
>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>> > >>>>>>>
>>>> > >>>>>>>> Hi,
>>>> > >>>>>>>>
>>>> > >>>>>>>> My point about "in real time application or data, there is
>>>> nothing
>>>> > >>>>>>>> as an answer which is supposed to be late and correct. The
>>>> timeliness is
>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>> slowly it becomes
>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need
>>>> this
>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>> > >>>>>>>>
>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of the
>>>> answer
>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>> streaming
>>>> > >>>>>>>> engine, primarily operating on micro-batches, often delivers
>>>> results that
>>>> > >>>>>>>> are technically "correct" but arrive too late to be truly
>>>> useful for
>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
>>>> "useless or
>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>> > >>>>>>>>
>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>> *high-frequency
>>>> > >>>>>>>> trading,* market data or trade execution commands must be
>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can mean
>>>> missed
>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>> "correct" price
>>>> > >>>>>>>> update useless if it's not instantaneous. able for these
>>>> demanding
>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply not
>>>> good enough. As
>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>> treated as such not
>>>> > >>>>>>>> as a comment.in SPIP
>>>> > >>>>>>>>
>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>> Analysis |
>>>> > >>>>>>>> GDPR
>>>> > >>>>>>>>
>>>> > >>>>>>>> view my Linkedin profile
>>>> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>> > >>>>>>>>
>>>> > >>>>>>>>
>>>> > >>>>>>>>
>>>> > >>>>>>>>
>>>> > >>>>>>>>
>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>> denny.g.lee@gmail.com>
>>>> > >>>>>>>> wrote:
>>>> > >>>>>>>>
>>>> > >>>>>>>>> Hey Mich,
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> Sorry, I may be missing something here but what does your
>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>>>> comments directly
>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet below is
>>>> a direct copy
>>>> > >>>>>>>>> from the SPIP itself.
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> Thanks,
>>>> > >>>>>>>>> Denny
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>> just to add
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>> definition of
>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>> application or
>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed to
>>>> be late and
>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if I
>>>> get the right
>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>> Analysis |
>>>> > >>>>>>>>>> GDPR
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> view my Linkedin profile
>>>> > >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>> micro-batching.If you
>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must
>>>> be balanced against
>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>> prevent back pressure
>>>> > >>>>>>>>>>> and instability. In the case of Continuous Processing
>>>> mode, a
>>>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>>>> interval quote
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> "
>>>> > >>>>>>>>>>> df.writeStream
>>>> > >>>>>>>>>>> .format("...")
>>>> > >>>>>>>>>>> .option("...")
>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
>>>> trigger
>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>> > >>>>>>>>>>> .start()
>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run
>>>> in the
>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval
>>>> can also be
>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each
>>>> micro-batch should
>>>> > >>>>>>>>>>> run for.
>>>> > >>>>>>>>>>> "
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>> > >>>>>>>>>>> HTH
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>> Analysis |
>>>> > >>>>>>>>>>> GDPR
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> view my Linkedin profile
>>>> > >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/
>>>> >
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>> Hi all,
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming”
>>>> that I've been
>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>>>> Jungtaek Lim, and
>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>> [Doc
>>>> > >>>>>>>>>>>> <
>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>> >
>>>> > >>>>>>>>>>>> ].
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time
>>>> Mode”
>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers
>>>> end-to-end latency
>>>> > >>>>>>>>>>>> for processing streams of data.
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our
>>>> goal is
>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that
>>>> need results almost
>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>>>> achieve this without
>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users
>>>> already use – so
>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>> ultra-low-latency mode by
>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>> personalization) that
>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>> Spark’s current streaming
>>>> > >>>>>>>>>>>> engine.
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>
>>>> > >>>>>> --
>>>> > >>>>>> Best,
>>>> > >>>>>> Yanbo
>>>> > >>>>>>
>>>> > >>>>>
>>>> > >>
>>>> > >> --
>>>> > >> John Zhuge
>>>> > >>
>>>> > >>
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jerry Peng <je...@gmail.com>.
Mark,
As an example of my point if you go the the Apache Storm (another stream
processing engine) website:
https://storm.apache.org/
It describes Storm as:
"Apache Storm is a free and open source distributed *realtime* computation
system"
If you can to apache Flink:
https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/
"Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing"
Thus, what the term "rea-time" implies in this should not be confusing for
folks in this area.
On Thu, May 29, 2025 at 10:22 PM Jerry Peng <je...@gmail.com>
wrote:
> Mich,
>
> If I understood your last email correctly, I think you also wanted to have
> a discussion about naming? Why are we calling this new execution mode
> described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
> "continuous mode" is taken and we want another name to describe an
> execution mode that provides ultra low latency processing. We could have
> called it "low latency mode", though I don't really like that naming since
> it implies the other execution modes are not low latency which I don't
> believe is true. This new proposed mode can simply deliver even lower
> latency. Thus, we came up with the name "Real-time Mode". Of course, we
> are talking about "soft" real-time here. I think when we are talking about
> distributed stream processing systems in the space of big data analytics,
> it is reasonable to assume anything described in this space as "real-time"
> implies "soft" real-time. Though if this is confusing or misleading, we
> can provide clear documentation on what "real-time" in real-time mode means
> and what it guarantees. Just my thoughts. I would love to hear other
> perspectives.
>
> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> I think from what I have seen there are a good number of +1 responses as
>> opposed to quantitative discussions (based on my observations only). Given
>> the objectives of the thread, we ought to focus on what is meant by real
>> time compared to continuous modes.To be fair, it is a common point of
>> confusion, and the terms are often used interchangeably in general
>> conversation, but in technical contexts, especially with streaming data
>> platforms, they have specific and important differences.
>>
>> "Continuous Mode" refers to a processing strategy that aims for true,
>> uninterrupted, sub-millisecond latency processing. Chiefly
>>
>> - Event-at-a-Time (or very small batch groups): The system processes
>> individual events or extremely small groups of events -> microbatches as
>> they flow through the pipeline.
>> - Minimal Latency: The primary goal is to achieve the absolute lowest
>> possible end-to-end latency, often in the order of milliseconds or even
>> below
>> - Most business use cases (say financial markets) can live with this
>> as they do not rely on rdges
>>
>> Now what is meant by "Real-time Mode"
>>
>> This is where the nuance comes in. "Real-time" is a broader and sometimes
>> more subjective term. When the text introduces "Real-time Mode" as distinct
>> from "Continuous Mode," it suggests a specific implementation that achieves
>> real-time characteristics but might do so differently or more robustly than
>> a "continuous" mode attempt. Going back to my earlier mention, in real time
>> application , there is nothing as an answer which is supposed to be late
>> and correct. The timeliness is part of the application. if I get the
>> right answer too slowly it becomes useless or wrong. What I call the "Late
>> and Correct is Useless" Principle
>>
>> In summary, "Real-time Mode" seems to describe an approach that delivers
>> low-latency processing with high reliability and ease of use, leveraging
>> established, battle-tested components.I invite the audience to have a
>> discussion on this.
>>
>> HTH
>>
>> Dr Mich Talebzadeh,
>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>>
>>> +1
>>>
>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>> > +1
>>> >
>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>> >
>>> > > +1.
>>> > >
>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
>>> > >
>>> > >> +1
>>> > >> Sent from my iPhone
>>> > >>
>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
>>> > >>
>>> > >>
>>> > >> +1 Nice feature
>>> > >>
>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xyliyuanjian@gmail.com
>>> >
>>> > >> wrote:
>>> > >>
>>> > >>> +1
>>> > >>>
>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>> > >>>
>>> > >>>> +1, LGTM.
>>> > >>>>
>>> > >>>> Kent
>>> > >>>>
>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>> > >>>>
>>> > >>>>> +1. Super excited by this initiative!
>>> > >>>>>
>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com>
>>> > >>>>> wrote:
>>> > >>>>>
>>> > >>>>>> +1
>>> > >>>>>>
>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>> huaxin.gao11@gmail.com>
>>> > >>>>>> wrote:
>>> > >>>>>>
>>> > >>>>>>> +1
>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>> > >>>>>>> eliminate the need for separate streaming engines, reducing
>>> system
>>> > >>>>>>> complexity and operational cost. Excited to see this direction!
>>> > >>>>>>>
>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>> > >>>>>>>
>>> > >>>>>>>> Hi,
>>> > >>>>>>>>
>>> > >>>>>>>> My point about "in real time application or data, there is
>>> nothing
>>> > >>>>>>>> as an answer which is supposed to be late and correct. The
>>> timeliness is
>>> > >>>>>>>> part of the application. if I get the right answer too slowly
>>> it becomes
>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need
>>> this
>>> > >>>>>>>> Spark Structured Streaming proposal.
>>> > >>>>>>>>
>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>> > >>>>>>>> applications where, as I define it, the *timeliness* of the
>>> answer
>>> > >>>>>>>> is as critical as its *correctness*. Spark's current streaming
>>> > >>>>>>>> engine, primarily operating on micro-batches, often delivers
>>> results that
>>> > >>>>>>>> are technically "correct" but arrive too late to be truly
>>> useful for
>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
>>> "useless or
>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>> > >>>>>>>>
>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>> *high-frequency
>>> > >>>>>>>> trading,* market data or trade execution commands must be
>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can mean
>>> missed
>>> > >>>>>>>> opportunities or significant financial losses, making a
>>> "correct" price
>>> > >>>>>>>> update useless if it's not instantaneous. able for these
>>> demanding
>>> > >>>>>>>> use cases, where a "late but correct" answer is simply not
>>> good enough. As
>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>> treated as such not
>>> > >>>>>>>> as a comment.in SPIP
>>> > >>>>>>>>
>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>> > >>>>>>>> Dr Mich Talebzadeh,
>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>> Analysis |
>>> > >>>>>>>> GDPR
>>> > >>>>>>>>
>>> > >>>>>>>> view my Linkedin profile
>>> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>> > >>>>>>>>
>>> > >>>>>>>>
>>> > >>>>>>>>
>>> > >>>>>>>>
>>> > >>>>>>>>
>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>> denny.g.lee@gmail.com>
>>> > >>>>>>>> wrote:
>>> > >>>>>>>>
>>> > >>>>>>>>> Hey Mich,
>>> > >>>>>>>>>
>>> > >>>>>>>>> Sorry, I may be missing something here but what does your
>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>>> comments directly
>>> > >>>>>>>>> to the SPIP to provide context as the code snippet below is
>>> a direct copy
>>> > >>>>>>>>> from the SPIP itself.
>>> > >>>>>>>>>
>>> > >>>>>>>>> Thanks,
>>> > >>>>>>>>> Denny
>>> > >>>>>>>>>
>>> > >>>>>>>>>
>>> > >>>>>>>>>
>>> > >>>>>>>>>
>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>> > >>>>>>>>>
>>> > >>>>>>>>>> just to add
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>> definition of
>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>> application or
>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed to be
>>> late and
>>> > >>>>>>>>>> correct. The timeliness is part of the application.if I get
>>> the right
>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>> Analysis |
>>> > >>>>>>>>>> GDPR
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> view my Linkedin profile
>>> > >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>> The current limitations in SSS come from micro-batching.If
>>> you
>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must be
>>> balanced against
>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>> prevent back pressure
>>> > >>>>>>>>>>> and instability. In the case of Continuous Processing
>>> mode, a
>>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>>> interval quote
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> "
>>> > >>>>>>>>>>> df.writeStream
>>> > >>>>>>>>>>> .format("...")
>>> > >>>>>>>>>>> .option("...")
>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
>>> trigger
>>> > >>>>>>>>>>> type to enable real-time Mode
>>> > >>>>>>>>>>> .start()
>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run in
>>> the
>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval can
>>> also be
>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each
>>> micro-batch should
>>> > >>>>>>>>>>> run for.
>>> > >>>>>>>>>>> "
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>> > >>>>>>>>>>> HTH
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>> Analysis |
>>> > >>>>>>>>>>> GDPR
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> view my Linkedin profile
>>> > >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>> Hi all,
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming”
>>> that I've been
>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>>> Jungtaek Lim, and
>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>> > >>>>>>>>>>>> <
>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>> >
>>> > >>>>>>>>>>>> ].
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time
>>> Mode”
>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers
>>> end-to-end latency
>>> > >>>>>>>>>>>> for processing streams of data.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our
>>> goal is
>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that
>>> need results almost
>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>>> achieve this without
>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users
>>> already use – so
>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>> ultra-low-latency mode by
>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>> personalization) that
>>> > >>>>>>>>>>>> today cannot meet their latency requirements with Spark’s
>>> current streaming
>>> > >>>>>>>>>>>> engine.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>> > >>>>>>>>>>>> suggestions on this approach!
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>
>>> > >>>>>>
>>> > >>>>>> --
>>> > >>>>>> Best,
>>> > >>>>>> Yanbo
>>> > >>>>>>
>>> > >>>>>
>>> > >>
>>> > >> --
>>> > >> John Zhuge
>>> > >>
>>> > >>
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jerry Peng <je...@gmail.com>.
Mich,
If I understood your last email correctly, I think you also wanted to have
a discussion about naming? Why are we calling this new execution mode
described in the SPIP "Real-time Mode"? Here are my two cents. Firstly,
"continuous mode" is taken and we want another name to describe an
execution mode that provides ultra low latency processing. We could have
called it "low latency mode", though I don't really like that naming since
it implies the other execution modes are not low latency which I don't
believe is true. This new proposed mode can simply deliver even lower
latency. Thus, we came up with the name "Real-time Mode". Of course, we
are talking about "soft" real-time here. I think when we are talking about
distributed stream processing systems in the space of big data analytics,
it is reasonable to assume anything described in this space as "real-time"
implies "soft" real-time. Though if this is confusing or misleading, we
can provide clear documentation on what "real-time" in real-time mode means
and what it guarantees. Just my thoughts. I would love to hear other
perspectives.
On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> I think from what I have seen there are a good number of +1 responses as
> opposed to quantitative discussions (based on my observations only). Given
> the objectives of the thread, we ought to focus on what is meant by real
> time compared to continuous modes.To be fair, it is a common point of
> confusion, and the terms are often used interchangeably in general
> conversation, but in technical contexts, especially with streaming data
> platforms, they have specific and important differences.
>
> "Continuous Mode" refers to a processing strategy that aims for true,
> uninterrupted, sub-millisecond latency processing. Chiefly
>
> - Event-at-a-Time (or very small batch groups): The system processes
> individual events or extremely small groups of events -> microbatches as
> they flow through the pipeline.
> - Minimal Latency: The primary goal is to achieve the absolute lowest
> possible end-to-end latency, often in the order of milliseconds or even
> below
> - Most business use cases (say financial markets) can live with this
> as they do not rely on rdges
>
> Now what is meant by "Real-time Mode"
>
> This is where the nuance comes in. "Real-time" is a broader and sometimes
> more subjective term. When the text introduces "Real-time Mode" as distinct
> from "Continuous Mode," it suggests a specific implementation that achieves
> real-time characteristics but might do so differently or more robustly than
> a "continuous" mode attempt. Going back to my earlier mention, in real time
> application , there is nothing as an answer which is supposed to be late
> and correct. The timeliness is part of the application. if I get the
> right answer too slowly it becomes useless or wrong. What I call the "Late
> and Correct is Useless" Principle
>
> In summary, "Real-time Mode" seems to describe an approach that delivers
> low-latency processing with high reliability and ease of use, leveraging
> established, battle-tested components.I invite the audience to have a
> discussion on this.
>
> HTH
>
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>
>> +1
>>
>> On 2025/05/29 16:25:19 Xiao Li wrote:
>> > +1
>> >
>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>> >
>> > > +1.
>> > >
>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
>> > >
>> > >> +1
>> > >> Sent from my iPhone
>> > >>
>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
>> > >>
>> > >>
>> > >> +1 Nice feature
>> > >>
>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xy...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> +1
>> > >>>
>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>> > >>>
>> > >>>> +1, LGTM.
>> > >>>>
>> > >>>> Kent
>> > >>>>
>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>> > >>>>
>> > >>>>> +1. Super excited by this initiative!
>> > >>>>>
>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> +1
>> > >>>>>>
>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>> huaxin.gao11@gmail.com>
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> +1
>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
>> > >>>>>>> eliminate the need for separate streaming engines, reducing
>> system
>> > >>>>>>> complexity and operational cost. Excited to see this direction!
>> > >>>>>>>
>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>> > >>>>>>>
>> > >>>>>>>> Hi,
>> > >>>>>>>>
>> > >>>>>>>> My point about "in real time application or data, there is
>> nothing
>> > >>>>>>>> as an answer which is supposed to be late and correct. The
>> timeliness is
>> > >>>>>>>> part of the application. if I get the right answer too slowly
>> it becomes
>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need this
>> > >>>>>>>> Spark Structured Streaming proposal.
>> > >>>>>>>>
>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>> > >>>>>>>> applications where, as I define it, the *timeliness* of the
>> answer
>> > >>>>>>>> is as critical as its *correctness*. Spark's current streaming
>> > >>>>>>>> engine, primarily operating on micro-batches, often delivers
>> results that
>> > >>>>>>>> are technically "correct" but arrive too late to be truly
>> useful for
>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
>> "useless or
>> > >>>>>>>> wrong" in a practical, business-critical sense.
>> > >>>>>>>>
>> > >>>>>>>> For example *in real-time fraud detection* and In
>> *high-frequency
>> > >>>>>>>> trading,* market data or trade execution commands must be
>> > >>>>>>>> delivered with minimal latency. Even a slight delay can mean
>> missed
>> > >>>>>>>> opportunities or significant financial losses, making a
>> "correct" price
>> > >>>>>>>> update useless if it's not instantaneous. able for these
>> demanding
>> > >>>>>>>> use cases, where a "late but correct" answer is simply not
>> good enough. As
>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>> treated as such not
>> > >>>>>>>> as a comment.in SPIP
>> > >>>>>>>>
>> > >>>>>>>> Hope this clarifies the connection in practical terms
>> > >>>>>>>> Dr Mich Talebzadeh,
>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis
>> |
>> > >>>>>>>> GDPR
>> > >>>>>>>>
>> > >>>>>>>> view my Linkedin profile
>> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <denny.g.lee@gmail.com
>> >
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hey Mich,
>> > >>>>>>>>>
>> > >>>>>>>>> Sorry, I may be missing something here but what does your
>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>> comments directly
>> > >>>>>>>>> to the SPIP to provide context as the code snippet below is a
>> direct copy
>> > >>>>>>>>> from the SPIP itself.
>> > >>>>>>>>>
>> > >>>>>>>>> Thanks,
>> > >>>>>>>>> Denny
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> just to add
>> > >>>>>>>>>>
>> > >>>>>>>>>> A stronger definition of real time. The engineering
>> definition of
>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>> > >>>>>>>>>>
>> > >>>>>>>>>> However, I put a stronger definition. In real time
>> application or
>> > >>>>>>>>>> data, there is nothing as an answer which is supposed to be
>> late and
>> > >>>>>>>>>> correct. The timeliness is part of the application.if I get
>> the right
>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> Dr Mich Talebzadeh,
>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>> Analysis |
>> > >>>>>>>>>> GDPR
>> > >>>>>>>>>>
>> > >>>>>>>>>> view my Linkedin profile
>> > >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>> The current limitations in SSS come from micro-batching.If
>> you
>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must be
>> balanced against
>> > >>>>>>>>>>> the available processing capacity of the cluster to prevent
>> back pressure
>> > >>>>>>>>>>> and instability. In the case of Continuous Processing mode,
>> a
>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>> interval quote
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> "
>> > >>>>>>>>>>> df.writeStream
>> > >>>>>>>>>>> .format("...")
>> > >>>>>>>>>>> .option("...")
>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
>> trigger
>> > >>>>>>>>>>> type to enable real-time Mode
>> > >>>>>>>>>>> .start()
>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run in
>> the
>> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval can
>> also be
>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each
>> micro-batch should
>> > >>>>>>>>>>> run for.
>> > >>>>>>>>>>> "
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
>> > >>>>>>>>>>> HTH
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>> Analysis |
>> > >>>>>>>>>>> GDPR
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> view my Linkedin profile
>> > >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>> Hi all,
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that
>> I've been
>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>> Jungtaek Lim, and
>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>> > >>>>>>>>>>>> <
>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>> >
>> > >>>>>>>>>>>> ].
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time
>> Mode”
>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers
>> end-to-end latency
>> > >>>>>>>>>>>> for processing streams of data.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our
>> goal is
>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that need
>> results almost
>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>> achieve this without
>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users
>> already use – so
>> > >>>>>>>>>>>> existing streaming queries can run in this new
>> ultra-low-latency mode by
>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>> personalization) that
>> > >>>>>>>>>>>> today cannot meet their latency requirements with Spark’s
>> current streaming
>> > >>>>>>>>>>>> engine.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>> > >>>>>>>>>>>> suggestions on this approach!
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>
>> > >>>>>> --
>> > >>>>>> Best,
>> > >>>>>> Yanbo
>> > >>>>>>
>> > >>>>>
>> > >>
>> > >> --
>> > >> John Zhuge
>> > >>
>> > >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mark Hamstra <ma...@gmail.com>.
It should not be assumed. In something called "real-time", it should be
very explicit what clock-time constraints are and are not guaranteed.
On Thu, May 29, 2025 at 10:00 PM Jerry Peng <je...@gmail.com>
wrote:
> It was kind of hard to see what mich's point was in the plethora of
> emails he sent :)
>
> In embedded systems, there is a concept of soft real-time and hard
> real-time. For these stream processing systems built for big data
> analytics, it is assumed that we are talking about soft real-time. Sure,
> there can be an argument as to why the name of this mode is not "low
> latency mode" but I honestly don't like naming. This naming implies the
> existing execution modes are not low latency which is not true. What
> defines "low" in low latency? It is kind of relative. That is why the name
> Real-time mode is selected.
>
> On Thu, May 29, 2025 at 8:57 PM Mark Hamstra <ma...@gmail.com>
> wrote:
>
>> I think you are missing his point. There is a fundamental difference
>> between low-latency computation and real-time computing. Is what is
>> described in the SPIP intended to provide results with real-time
>> guarantees, or is it a misnamed effort to achieve low-latency?
>>
>> On Thu, May 29, 2025 at 5:54 PM Jerry Peng <je...@gmail.com>
>> wrote:
>>
>>> Mich,
>>>
>>> Thank you for chiming in and providing insights into the importance of
>>> not only getting correct results but also timely results. You are
>>> absolutely right that the reason why something like Real-time Mode is
>>> valuable is its ability to provide timely results for certain use cases
>>> that require users to react super quickly to data. I can emphasize this
>>> point in the SPIP.
>>>
>>>
>>> On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> I think from what I have seen there are a good number of +1 responses
>>>> as opposed to quantitative discussions (based on my observations only).
>>>> Given the objectives of the thread, we ought to focus on what is meant by
>>>> real time compared to continuous modes.To be fair, it is a common point
>>>> of confusion, and the terms are often used interchangeably in general
>>>> conversation, but in technical contexts, especially with streaming data
>>>> platforms, they have specific and important differences.
>>>>
>>>> "Continuous Mode" refers to a processing strategy that aims for true,
>>>> uninterrupted, sub-millisecond latency processing. Chiefly
>>>>
>>>> - Event-at-a-Time (or very small batch groups): The system
>>>> processes individual events or extremely small groups of events ->
>>>> microbatches as they flow through the pipeline.
>>>> - Minimal Latency: The primary goal is to achieve the absolute
>>>> lowest possible end-to-end latency, often in the order of milliseconds or
>>>> even below
>>>> - Most business use cases (say financial markets) can live with
>>>> this as they do not rely on rdges
>>>>
>>>> Now what is meant by "Real-time Mode"
>>>>
>>>> This is where the nuance comes in. "Real-time" is a broader and
>>>> sometimes more subjective term. When the text introduces "Real-time Mode"
>>>> as distinct from "Continuous Mode," it suggests a specific implementation
>>>> that achieves real-time characteristics but might do so differently or more
>>>> robustly than a "continuous" mode attempt. Going back to my earlier
>>>> mention, in real time application , there is nothing as an answer which is
>>>> supposed to be late and correct. The timeliness is part of the application.
>>>> if I get the right answer too slowly it becomes useless or wrong. What I
>>>> call the "Late and Correct is Useless" Principle
>>>>
>>>> In summary, "Real-time Mode" seems to describe an approach that
>>>> delivers low-latency processing with high reliability and ease of use,
>>>> leveraging established, battle-tested components.I invite the audience to
>>>> have a discussion on this.
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh,
>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On 2025/05/29 16:25:19 Xiao Li wrote:
>>>>> > +1
>>>>> >
>>>>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>>>>> >
>>>>> > > +1.
>>>>> > >
>>>>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
>>>>> > >
>>>>> > >> +1
>>>>> > >> Sent from my iPhone
>>>>> > >>
>>>>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org>
>>>>> wrote:
>>>>> > >>
>>>>> > >>
>>>>> > >> +1 Nice feature
>>>>> > >>
>>>>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <
>>>>> xyliyuanjian@gmail.com>
>>>>> > >> wrote:
>>>>> > >>
>>>>> > >>> +1
>>>>> > >>>
>>>>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>> > >>>
>>>>> > >>>> +1, LGTM.
>>>>> > >>>>
>>>>> > >>>> Kent
>>>>> > >>>>
>>>>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>> > >>>>
>>>>> > >>>>> +1. Super excited by this initiative!
>>>>> > >>>>>
>>>>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <
>>>>> ybliang8@gmail.com>
>>>>> > >>>>> wrote:
>>>>> > >>>>>
>>>>> > >>>>>> +1
>>>>> > >>>>>>
>>>>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>>>>> huaxin.gao11@gmail.com>
>>>>> > >>>>>> wrote:
>>>>> > >>>>>>
>>>>> > >>>>>>> +1
>>>>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>>>> > >>>>>>> eliminate the need for separate streaming engines, reducing
>>>>> system
>>>>> > >>>>>>> complexity and operational cost. Excited to see this
>>>>> direction!
>>>>> > >>>>>>>
>>>>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>> > >>>>>>>
>>>>> > >>>>>>>> Hi,
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> My point about "in real time application or data, there is
>>>>> nothing
>>>>> > >>>>>>>> as an answer which is supposed to be late and correct. The
>>>>> timeliness is
>>>>> > >>>>>>>> part of the application. if I get the right answer too
>>>>> slowly it becomes
>>>>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need
>>>>> this
>>>>> > >>>>>>>> Spark Structured Streaming proposal.
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>>>>> > >>>>>>>> applications where, as I define it, the *timeliness* of the
>>>>> answer
>>>>> > >>>>>>>> is as critical as its *correctness*. Spark's current
>>>>> streaming
>>>>> > >>>>>>>> engine, primarily operating on micro-batches, often
>>>>> delivers results that
>>>>> > >>>>>>>> are technically "correct" but arrive too late to be truly
>>>>> useful for
>>>>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
>>>>> "useless or
>>>>> > >>>>>>>> wrong" in a practical, business-critical sense.
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> For example *in real-time fraud detection* and In
>>>>> *high-frequency
>>>>> > >>>>>>>> trading,* market data or trade execution commands must be
>>>>> > >>>>>>>> delivered with minimal latency. Even a slight delay can
>>>>> mean missed
>>>>> > >>>>>>>> opportunities or significant financial losses, making a
>>>>> "correct" price
>>>>> > >>>>>>>> update useless if it's not instantaneous. able for these
>>>>> demanding
>>>>> > >>>>>>>> use cases, where a "late but correct" answer is simply not
>>>>> good enough. As
>>>>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>>>>> treated as such not
>>>>> > >>>>>>>> as a comment.in SPIP
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> Hope this clarifies the connection in practical terms
>>>>> > >>>>>>>> Dr Mich Talebzadeh,
>>>>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>> Analysis |
>>>>> > >>>>>>>> GDPR
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> view my Linkedin profile
>>>>> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <
>>>>> denny.g.lee@gmail.com>
>>>>> > >>>>>>>> wrote:
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>> Hey Mich,
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Sorry, I may be missing something here but what does your
>>>>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>>>>> comments directly
>>>>> > >>>>>>>>> to the SPIP to provide context as the code snippet below
>>>>> is a direct copy
>>>>> > >>>>>>>>> from the SPIP itself.
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Thanks,
>>>>> > >>>>>>>>> Denny
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>> just to add
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> A stronger definition of real time. The engineering
>>>>> definition of
>>>>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> However, I put a stronger definition. In real time
>>>>> application or
>>>>> > >>>>>>>>>> data, there is nothing as an answer which is supposed to
>>>>> be late and
>>>>> > >>>>>>>>>> correct. The timeliness is part of the application.if I
>>>>> get the right
>>>>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> Dr Mich Talebzadeh,
>>>>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>> Analysis |
>>>>> > >>>>>>>>>> GDPR
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> view my Linkedin profile
>>>>> > >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/
>>>>> >
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>> The current limitations in SSS come from
>>>>> micro-batching.If you
>>>>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must
>>>>> be balanced against
>>>>> > >>>>>>>>>>> the available processing capacity of the cluster to
>>>>> prevent back pressure
>>>>> > >>>>>>>>>>> and instability. In the case of Continuous Processing
>>>>> mode, a
>>>>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>>>>> interval quote
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> "
>>>>> > >>>>>>>>>>> df.writeStream
>>>>> > >>>>>>>>>>> .format("...")
>>>>> > >>>>>>>>>>> .option("...")
>>>>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
>>>>> trigger
>>>>> > >>>>>>>>>>> type to enable real-time Mode
>>>>> > >>>>>>>>>>> .start()
>>>>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run
>>>>> in the
>>>>> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval
>>>>> can also be
>>>>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each
>>>>> micro-batch should
>>>>> > >>>>>>>>>>> run for.
>>>>> > >>>>>>>>>>> "
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>> > >>>>>>>>>>> HTH
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>>>>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>>>>> Analysis |
>>>>> > >>>>>>>>>>> GDPR
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> view my Linkedin profile
>>>>> > >>>>>>>>>>> <
>>>>> https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>> Hi all,
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming”
>>>>> that I've been
>>>>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>>>>> Jungtaek Lim, and
>>>>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>]
>>>>> [Doc
>>>>> > >>>>>>>>>>>> <
>>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>>>>> >
>>>>> > >>>>>>>>>>>> ].
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called
>>>>> “Real-time Mode”
>>>>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers
>>>>> end-to-end latency
>>>>> > >>>>>>>>>>>> for processing streams of data.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our
>>>>> goal is
>>>>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that
>>>>> need results almost
>>>>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>>>>> achieve this without
>>>>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that
>>>>> users already use – so
>>>>> > >>>>>>>>>>>> existing streaming queries can run in this new
>>>>> ultra-low-latency mode by
>>>>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power
>>>>> real-time
>>>>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>>>>> personalization) that
>>>>> > >>>>>>>>>>>> today cannot meet their latency requirements with
>>>>> Spark’s current streaming
>>>>> > >>>>>>>>>>>> engine.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>> > >>>>>>>>>>>> suggestions on this approach!
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>
>>>>> > >>>>>> --
>>>>> > >>>>>> Best,
>>>>> > >>>>>> Yanbo
>>>>> > >>>>>>
>>>>> > >>>>>
>>>>> > >>
>>>>> > >> --
>>>>> > >> John Zhuge
>>>>> > >>
>>>>> > >>
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jerry Peng <je...@gmail.com>.
Mich,
Thank you for chiming in and providing insights into the importance of not
only getting correct results but also timely results. You are absolutely
right that the reason why something like Real-time Mode is valuable is its
ability to provide timely results for certain use cases that require users
to react super quickly to data. I can emphasize this point in the SPIP.
On Thu, May 29, 2025 at 3:48 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> I think from what I have seen there are a good number of +1 responses as
> opposed to quantitative discussions (based on my observations only). Given
> the objectives of the thread, we ought to focus on what is meant by real
> time compared to continuous modes.To be fair, it is a common point of
> confusion, and the terms are often used interchangeably in general
> conversation, but in technical contexts, especially with streaming data
> platforms, they have specific and important differences.
>
> "Continuous Mode" refers to a processing strategy that aims for true,
> uninterrupted, sub-millisecond latency processing. Chiefly
>
> - Event-at-a-Time (or very small batch groups): The system processes
> individual events or extremely small groups of events -> microbatches as
> they flow through the pipeline.
> - Minimal Latency: The primary goal is to achieve the absolute lowest
> possible end-to-end latency, often in the order of milliseconds or even
> below
> - Most business use cases (say financial markets) can live with this
> as they do not rely on rdges
>
> Now what is meant by "Real-time Mode"
>
> This is where the nuance comes in. "Real-time" is a broader and sometimes
> more subjective term. When the text introduces "Real-time Mode" as distinct
> from "Continuous Mode," it suggests a specific implementation that achieves
> real-time characteristics but might do so differently or more robustly than
> a "continuous" mode attempt. Going back to my earlier mention, in real time
> application , there is nothing as an answer which is supposed to be late
> and correct. The timeliness is part of the application. if I get the
> right answer too slowly it becomes useless or wrong. What I call the "Late
> and Correct is Useless" Principle
>
> In summary, "Real-time Mode" seems to describe an approach that delivers
> low-latency processing with high reliability and ease of use, leveraging
> established, battle-tested components.I invite the audience to have a
> discussion on this.
>
> HTH
>
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
>
>> +1
>>
>> On 2025/05/29 16:25:19 Xiao Li wrote:
>> > +1
>> >
>> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>> >
>> > > +1.
>> > >
>> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
>> > >
>> > >> +1
>> > >> Sent from my iPhone
>> > >>
>> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
>> > >>
>> > >>
>> > >> +1 Nice feature
>> > >>
>> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xy...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> +1
>> > >>>
>> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>> > >>>
>> > >>>> +1, LGTM.
>> > >>>>
>> > >>>> Kent
>> > >>>>
>> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>> > >>>>
>> > >>>>> +1. Super excited by this initiative!
>> > >>>>>
>> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> +1
>> > >>>>>>
>> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
>> huaxin.gao11@gmail.com>
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> +1
>> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
>> > >>>>>>> eliminate the need for separate streaming engines, reducing
>> system
>> > >>>>>>> complexity and operational cost. Excited to see this direction!
>> > >>>>>>>
>> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
>> > >>>>>>>
>> > >>>>>>>> Hi,
>> > >>>>>>>>
>> > >>>>>>>> My point about "in real time application or data, there is
>> nothing
>> > >>>>>>>> as an answer which is supposed to be late and correct. The
>> timeliness is
>> > >>>>>>>> part of the application. if I get the right answer too slowly
>> it becomes
>> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need this
>> > >>>>>>>> Spark Structured Streaming proposal.
>> > >>>>>>>>
>> > >>>>>>>> The proposal is precisely about enabling Spark to power
>> > >>>>>>>> applications where, as I define it, the *timeliness* of the
>> answer
>> > >>>>>>>> is as critical as its *correctness*. Spark's current streaming
>> > >>>>>>>> engine, primarily operating on micro-batches, often delivers
>> results that
>> > >>>>>>>> are technically "correct" but arrive too late to be truly
>> useful for
>> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
>> "useless or
>> > >>>>>>>> wrong" in a practical, business-critical sense.
>> > >>>>>>>>
>> > >>>>>>>> For example *in real-time fraud detection* and In
>> *high-frequency
>> > >>>>>>>> trading,* market data or trade execution commands must be
>> > >>>>>>>> delivered with minimal latency. Even a slight delay can mean
>> missed
>> > >>>>>>>> opportunities or significant financial losses, making a
>> "correct" price
>> > >>>>>>>> update useless if it's not instantaneous. able for these
>> demanding
>> > >>>>>>>> use cases, where a "late but correct" answer is simply not
>> good enough. As
>> > >>>>>>>> a colliery it is a fundamental concept, so it has to be
>> treated as such not
>> > >>>>>>>> as a comment.in SPIP
>> > >>>>>>>>
>> > >>>>>>>> Hope this clarifies the connection in practical terms
>> > >>>>>>>> Dr Mich Talebzadeh,
>> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis
>> |
>> > >>>>>>>> GDPR
>> > >>>>>>>>
>> > >>>>>>>> view my Linkedin profile
>> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <denny.g.lee@gmail.com
>> >
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hey Mich,
>> > >>>>>>>>>
>> > >>>>>>>>> Sorry, I may be missing something here but what does your
>> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
>> comments directly
>> > >>>>>>>>> to the SPIP to provide context as the code snippet below is a
>> direct copy
>> > >>>>>>>>> from the SPIP itself.
>> > >>>>>>>>>
>> > >>>>>>>>> Thanks,
>> > >>>>>>>>> Denny
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> just to add
>> > >>>>>>>>>>
>> > >>>>>>>>>> A stronger definition of real time. The engineering
>> definition of
>> > >>>>>>>>>> real time is roughly fast enough to be interactive
>> > >>>>>>>>>>
>> > >>>>>>>>>> However, I put a stronger definition. In real time
>> application or
>> > >>>>>>>>>> data, there is nothing as an answer which is supposed to be
>> late and
>> > >>>>>>>>>> correct. The timeliness is part of the application.if I get
>> the right
>> > >>>>>>>>>> answer too slowly it becomes useless or wrong
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> Dr Mich Talebzadeh,
>> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>> Analysis |
>> > >>>>>>>>>> GDPR
>> > >>>>>>>>>>
>> > >>>>>>>>>> view my Linkedin profile
>> > >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>> The current limitations in SSS come from micro-batching.If
>> you
>> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must be
>> balanced against
>> > >>>>>>>>>>> the available processing capacity of the cluster to prevent
>> back pressure
>> > >>>>>>>>>>> and instability. In the case of Continuous Processing mode,
>> a
>> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
>> interval quote
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> "
>> > >>>>>>>>>>> df.writeStream
>> > >>>>>>>>>>> .format("...")
>> > >>>>>>>>>>> .option("...")
>> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
>> trigger
>> > >>>>>>>>>>> type to enable real-time Mode
>> > >>>>>>>>>>> .start()
>> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run in
>> the
>> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval can
>> also be
>> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each
>> micro-batch should
>> > >>>>>>>>>>> run for.
>> > >>>>>>>>>>> "
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
>> > >>>>>>>>>>> HTH
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Dr Mich Talebzadeh,
>> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
>> Analysis |
>> > >>>>>>>>>>> GDPR
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> view my Linkedin profile
>> > >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>> Hi all,
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that
>> I've been
>> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
>> Jungtaek Lim, and
>> > >>>>>>>>>>>> Michael Armbrust: [JIRA
>> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>> > >>>>>>>>>>>> <
>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
>> >
>> > >>>>>>>>>>>> ].
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time
>> Mode”
>> > >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers
>> end-to-end latency
>> > >>>>>>>>>>>> for processing streams of data.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our
>> goal is
>> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that need
>> results almost
>> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
>> achieve this without
>> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users
>> already use – so
>> > >>>>>>>>>>>> existing streaming queries can run in this new
>> ultra-low-latency mode by
>> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
>> personalization) that
>> > >>>>>>>>>>>> today cannot meet their latency requirements with Spark’s
>> current streaming
>> > >>>>>>>>>>>> engine.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>> > >>>>>>>>>>>> suggestions on this approach!
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>
>> > >>>>>> --
>> > >>>>>> Best,
>> > >>>>>> Yanbo
>> > >>>>>>
>> > >>>>>
>> > >>
>> > >> --
>> > >> John Zhuge
>> > >>
>> > >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mich Talebzadeh <mi...@gmail.com>.
I think from what I have seen there are a good number of +1 responses as
opposed to quantitative discussions (based on my observations only). Given
the objectives of the thread, we ought to focus on what is meant by real
time compared to continuous modes.To be fair, it is a common point of
confusion, and the terms are often used interchangeably in general
conversation, but in technical contexts, especially with streaming data
platforms, they have specific and important differences.
"Continuous Mode" refers to a processing strategy that aims for true,
uninterrupted, sub-millisecond latency processing. Chiefly
- Event-at-a-Time (or very small batch groups): The system processes
individual events or extremely small groups of events -> microbatches as
they flow through the pipeline.
- Minimal Latency: The primary goal is to achieve the absolute lowest
possible end-to-end latency, often in the order of milliseconds or even
below
- Most business use cases (say financial markets) can live with this as
they do not rely on rdges
Now what is meant by "Real-time Mode"
This is where the nuance comes in. "Real-time" is a broader and sometimes
more subjective term. When the text introduces "Real-time Mode" as distinct
from "Continuous Mode," it suggests a specific implementation that achieves
real-time characteristics but might do so differently or more robustly than
a "continuous" mode attempt. Going back to my earlier mention, in real time
application , there is nothing as an answer which is supposed to be late
and correct. The timeliness is part of the application. if I get the
right answer too slowly it becomes useless or wrong. What I call the "Late
and Correct is Useless" Principle
In summary, "Real-time Mode" seems to describe an approach that delivers
low-latency processing with high reliability and ease of use, leveraging
established, battle-tested components.I invite the audience to have a
discussion on this.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Thu, 29 May 2025 at 19:15, Yang Jie <ya...@apache.org> wrote:
> +1
>
> On 2025/05/29 16:25:19 Xiao Li wrote:
> > +1
> >
> > Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
> >
> > > +1.
> > >
> > > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
> > >
> > >> +1
> > >> Sent from my iPhone
> > >>
> > >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
> > >>
> > >>
> > >> +1 Nice feature
> > >>
> > >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xy...@gmail.com>
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
> > >>>
> > >>>> +1, LGTM.
> > >>>>
> > >>>> Kent
> > >>>>
> > >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
> > >>>>
> > >>>>> +1. Super excited by this initiative!
> > >>>>>
> > >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> +1
> > >>>>>>
> > >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <
> huaxin.gao11@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> +1
> > >>>>>>> By unifying batch and low-latency streaming in Spark, we can
> > >>>>>>> eliminate the need for separate streaming engines, reducing
> system
> > >>>>>>> complexity and operational cost. Excited to see this direction!
> > >>>>>>>
> > >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
> > >>>>>>> mich.talebzadeh@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> My point about "in real time application or data, there is
> nothing
> > >>>>>>>> as an answer which is supposed to be late and correct. The
> timeliness is
> > >>>>>>>> part of the application. if I get the right answer too slowly
> it becomes
> > >>>>>>>> useless or wrong" is actually fundamental to *why* we need this
> > >>>>>>>> Spark Structured Streaming proposal.
> > >>>>>>>>
> > >>>>>>>> The proposal is precisely about enabling Spark to power
> > >>>>>>>> applications where, as I define it, the *timeliness* of the
> answer
> > >>>>>>>> is as critical as its *correctness*. Spark's current streaming
> > >>>>>>>> engine, primarily operating on micro-batches, often delivers
> results that
> > >>>>>>>> are technically "correct" but arrive too late to be truly
> useful for
> > >>>>>>>> certain high-stakes, real-time scenarios. This makes them
> "useless or
> > >>>>>>>> wrong" in a practical, business-critical sense.
> > >>>>>>>>
> > >>>>>>>> For example *in real-time fraud detection* and In
> *high-frequency
> > >>>>>>>> trading,* market data or trade execution commands must be
> > >>>>>>>> delivered with minimal latency. Even a slight delay can mean
> missed
> > >>>>>>>> opportunities or significant financial losses, making a
> "correct" price
> > >>>>>>>> update useless if it's not instantaneous. able for these
> demanding
> > >>>>>>>> use cases, where a "late but correct" answer is simply not good
> enough. As
> > >>>>>>>> a colliery it is a fundamental concept, so it has to be treated
> as such not
> > >>>>>>>> as a comment.in SPIP
> > >>>>>>>>
> > >>>>>>>> Hope this clarifies the connection in practical terms
> > >>>>>>>> Dr Mich Talebzadeh,
> > >>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
> > >>>>>>>> GDPR
> > >>>>>>>>
> > >>>>>>>> view my Linkedin profile
> > >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hey Mich,
> > >>>>>>>>>
> > >>>>>>>>> Sorry, I may be missing something here but what does your
> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add
> comments directly
> > >>>>>>>>> to the SPIP to provide context as the code snippet below is a
> direct copy
> > >>>>>>>>> from the SPIP itself.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Denny
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
> > >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> just to add
> > >>>>>>>>>>
> > >>>>>>>>>> A stronger definition of real time. The engineering
> definition of
> > >>>>>>>>>> real time is roughly fast enough to be interactive
> > >>>>>>>>>>
> > >>>>>>>>>> However, I put a stronger definition. In real time
> application or
> > >>>>>>>>>> data, there is nothing as an answer which is supposed to be
> late and
> > >>>>>>>>>> correct. The timeliness is part of the application.if I get
> the right
> > >>>>>>>>>> answer too slowly it becomes useless or wrong
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Dr Mich Talebzadeh,
> > >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
> Analysis |
> > >>>>>>>>>> GDPR
> > >>>>>>>>>>
> > >>>>>>>>>> view my Linkedin profile
> > >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
> > >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> The current limitations in SSS come from micro-batching.If
> you
> > >>>>>>>>>>> are going to reduce micro-batching, this reduction must be
> balanced against
> > >>>>>>>>>>> the available processing capacity of the cluster to prevent
> back pressure
> > >>>>>>>>>>> and instability. In the case of Continuous Processing mode, a
> > >>>>>>>>>>> specific continuous trigger with a desired checkpoint
> interval quote
> > >>>>>>>>>>>
> > >>>>>>>>>>> "
> > >>>>>>>>>>> df.writeStream
> > >>>>>>>>>>> .format("...")
> > >>>>>>>>>>> .option("...")
> > >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new
> trigger
> > >>>>>>>>>>> type to enable real-time Mode
> > >>>>>>>>>>> .start()
> > >>>>>>>>>>> This Trigger.RealTime signals that the query should run in
> the
> > >>>>>>>>>>> new ultra low-latency execution mode. A time interval can
> also be
> > >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each
> micro-batch should
> > >>>>>>>>>>> run for.
> > >>>>>>>>>>> "
> > >>>>>>>>>>>
> > >>>>>>>>>>> will inevitably depend on many factors. Not that simple
> > >>>>>>>>>>> HTH
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Dr Mich Talebzadeh,
> > >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic
> Analysis |
> > >>>>>>>>>>> GDPR
> > >>>>>>>>>>>
> > >>>>>>>>>>> view my Linkedin profile
> > >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
> > >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
> > >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that
> I've been
> > >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun,
> Jungtaek Lim, and
> > >>>>>>>>>>>> Michael Armbrust: [JIRA
> > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
> > >>>>>>>>>>>> <
> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing
> >
> > >>>>>>>>>>>> ].
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time
> Mode”
> > >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers
> end-to-end latency
> > >>>>>>>>>>>> for processing streams of data.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> A key principle of this proposal is compatibility. Our goal
> is
> > >>>>>>>>>>>> to make Spark capable of handling streaming jobs that need
> results almost
> > >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to
> achieve this without
> > >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users
> already use – so
> > >>>>>>>>>>>> existing streaming queries can run in this new
> ultra-low-latency mode by
> > >>>>>>>>>>>> simply turning it on, without rewriting their logic.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
> > >>>>>>>>>>>> applications (like instant anomaly alerts or live
> personalization) that
> > >>>>>>>>>>>> today cannot meet their latency requirements with Spark’s
> current streaming
> > >>>>>>>>>>>> engine.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
> > >>>>>>>>>>>> suggestions on this approach!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best,
> > >>>>>> Yanbo
> > >>>>>>
> > >>>>>
> > >>
> > >> --
> > >> John Zhuge
> > >>
> > >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Yang Jie <ya...@apache.org>.
+1
On 2025/05/29 16:25:19 Xiao Li wrote:
> +1
>
> Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>
> > +1.
> >
> > On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
> >
> >> +1
> >> Sent from my iPhone
> >>
> >> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
> >>
> >>
> >> +1 Nice feature
> >>
> >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xy...@gmail.com>
> >> wrote:
> >>
> >>> +1
> >>>
> >>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
> >>>
> >>>> +1, LGTM.
> >>>>
> >>>> Kent
> >>>>
> >>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
> >>>>
> >>>>> +1. Super excited by this initiative!
> >>>>>
> >>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> +1
> >>>>>>> By unifying batch and low-latency streaming in Spark, we can
> >>>>>>> eliminate the need for separate streaming engines, reducing system
> >>>>>>> complexity and operational cost. Excited to see this direction!
> >>>>>>>
> >>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
> >>>>>>> mich.talebzadeh@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> My point about "in real time application or data, there is nothing
> >>>>>>>> as an answer which is supposed to be late and correct. The timeliness is
> >>>>>>>> part of the application. if I get the right answer too slowly it becomes
> >>>>>>>> useless or wrong" is actually fundamental to *why* we need this
> >>>>>>>> Spark Structured Streaming proposal.
> >>>>>>>>
> >>>>>>>> The proposal is precisely about enabling Spark to power
> >>>>>>>> applications where, as I define it, the *timeliness* of the answer
> >>>>>>>> is as critical as its *correctness*. Spark's current streaming
> >>>>>>>> engine, primarily operating on micro-batches, often delivers results that
> >>>>>>>> are technically "correct" but arrive too late to be truly useful for
> >>>>>>>> certain high-stakes, real-time scenarios. This makes them "useless or
> >>>>>>>> wrong" in a practical, business-critical sense.
> >>>>>>>>
> >>>>>>>> For example *in real-time fraud detection* and In *high-frequency
> >>>>>>>> trading,* market data or trade execution commands must be
> >>>>>>>> delivered with minimal latency. Even a slight delay can mean missed
> >>>>>>>> opportunities or significant financial losses, making a "correct" price
> >>>>>>>> update useless if it's not instantaneous. able for these demanding
> >>>>>>>> use cases, where a "late but correct" answer is simply not good enough. As
> >>>>>>>> a colliery it is a fundamental concept, so it has to be treated as such not
> >>>>>>>> as a comment.in SPIP
> >>>>>>>>
> >>>>>>>> Hope this clarifies the connection in practical terms
> >>>>>>>> Dr Mich Talebzadeh,
> >>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
> >>>>>>>> GDPR
> >>>>>>>>
> >>>>>>>> view my Linkedin profile
> >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hey Mich,
> >>>>>>>>>
> >>>>>>>>> Sorry, I may be missing something here but what does your
> >>>>>>>>> definition here have to do with the SPIP? Perhaps add comments directly
> >>>>>>>>> to the SPIP to provide context as the code snippet below is a direct copy
> >>>>>>>>> from the SPIP itself.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Denny
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
> >>>>>>>>> mich.talebzadeh@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> just to add
> >>>>>>>>>>
> >>>>>>>>>> A stronger definition of real time. The engineering definition of
> >>>>>>>>>> real time is roughly fast enough to be interactive
> >>>>>>>>>>
> >>>>>>>>>> However, I put a stronger definition. In real time application or
> >>>>>>>>>> data, there is nothing as an answer which is supposed to be late and
> >>>>>>>>>> correct. The timeliness is part of the application.if I get the right
> >>>>>>>>>> answer too slowly it becomes useless or wrong
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Dr Mich Talebzadeh,
> >>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
> >>>>>>>>>> GDPR
> >>>>>>>>>>
> >>>>>>>>>> view my Linkedin profile
> >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
> >>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> The current limitations in SSS come from micro-batching.If you
> >>>>>>>>>>> are going to reduce micro-batching, this reduction must be balanced against
> >>>>>>>>>>> the available processing capacity of the cluster to prevent back pressure
> >>>>>>>>>>> and instability. In the case of Continuous Processing mode, a
> >>>>>>>>>>> specific continuous trigger with a desired checkpoint interval quote
> >>>>>>>>>>>
> >>>>>>>>>>> "
> >>>>>>>>>>> df.writeStream
> >>>>>>>>>>> .format("...")
> >>>>>>>>>>> .option("...")
> >>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger
> >>>>>>>>>>> type to enable real-time Mode
> >>>>>>>>>>> .start()
> >>>>>>>>>>> This Trigger.RealTime signals that the query should run in the
> >>>>>>>>>>> new ultra low-latency execution mode. A time interval can also be
> >>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each micro-batch should
> >>>>>>>>>>> run for.
> >>>>>>>>>>> "
> >>>>>>>>>>>
> >>>>>>>>>>> will inevitably depend on many factors. Not that simple
> >>>>>>>>>>> HTH
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Dr Mich Talebzadeh,
> >>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
> >>>>>>>>>>> GDPR
> >>>>>>>>>>>
> >>>>>>>>>>> view my Linkedin profile
> >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
> >>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
> >>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that I've been
> >>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and
> >>>>>>>>>>>> Michael Armbrust: [JIRA
> >>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
> >>>>>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
> >>>>>>>>>>>> ].
> >>>>>>>>>>>>
> >>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode”
> >>>>>>>>>>>> in Spark Structured Streaming that significantly lowers end-to-end latency
> >>>>>>>>>>>> for processing streams of data.
> >>>>>>>>>>>>
> >>>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is
> >>>>>>>>>>>> to make Spark capable of handling streaming jobs that need results almost
> >>>>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
> >>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
> >>>>>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
> >>>>>>>>>>>> simply turning it on, without rewriting their logic.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
> >>>>>>>>>>>> applications (like instant anomaly alerts or live personalization) that
> >>>>>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
> >>>>>>>>>>>> engine.
> >>>>>>>>>>>>
> >>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
> >>>>>>>>>>>> suggestions on this approach!
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best,
> >>>>>> Yanbo
> >>>>>>
> >>>>>
> >>
> >> --
> >> John Zhuge
> >>
> >>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Jerry Peng <je...@gmail.com>.
Hi all,
A big thanks to everyone that provided feedback to the SPIP! My co-authors
and I really appreciate it. I am excited to see this amount of interest in
the proposal. I am also glad to see all the support this initiative is
getting from the community.
Let me summarize some of the common questions and concerns folks commented
on in the SPIP google doc.
Common questions/concerns:
Why do you think Real-time mode will be more successful than Continuous
Mode?
I think there are a couple reasons that separate real-time mode and
continuous mode:
1.
Clear plan to support both stateless and stateful queries
2.
Reusing existing battle tested Structured Streaming components (e.g.,
checkpointing, offset tracking, etc).
3.
Broad functionality support so that users can easily switch between
real-time mode and other execution modes. This lowers the barrier to entry
for using real-time mode.
Is Real-time Mode going to replace Micro-batch Mode?
The existing execution modes i.e. ProcessingTime and AvailableNow still
have their own merits. It is easy to control the cost when running queries
in those modes. It is conceivable that some use cases may migrate to
predominately use real-time mode though I would imagine cost sensitive /
prioritized workloads will remain on the existing execution modes.
What are the processing guarantees and failure semantics for Real-time Mode?
The processing semantics remains exactly-once. This means updates to
internally managed state will only be reflected once.
In Real-time mode when a task fails, we will fail the query instead of
retrying the task. This needs to happen for queries that contain a
"streaming shuffle" i.e. multi-stage queries. The streaming shuffle does
not persist shuffle data, it simply sends data point to point, thus tasks
that read from a shuffle cannot simply restart. In the future we can
explore shuffle mechanisms that persist the shuffle data, e.g. persistent
queue, but something like that will incur additional infrastructure costs.
How will watermarks be updated in Real-time Mode?
We plan to build a new watermark propagation mechanism so watermark or
control messages can flow interleaved with data messages in the query. This
will allow event time information to reach operators sooner than batch
boundaries. A separate design document will be created to explain the
details. Please note that this feature is only really needed for
functionality that depends on advancement of event time to output results,
such as queries running in Append Mode. Best,
Jerry
On Thu, May 29, 2025 at 9:25 AM Xiao Li <ga...@gmail.com> wrote:
> +1
>
> Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
>
>> +1.
>>
>> On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
>>
>>> +1
>>> Sent from my iPhone
>>>
>>> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
>>>
>>>
>>> +1 Nice feature
>>>
>>> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xy...@gmail.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>>
>>>>> +1, LGTM.
>>>>>
>>>>> Kent
>>>>>
>>>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>>
>>>>>> +1. Super excited by this initiative!
>>>>>>
>>>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>>>>>>> eliminate the need for separate streaming engines, reducing system
>>>>>>>> complexity and operational cost. Excited to see this direction!
>>>>>>>>
>>>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> My point about "in real time application or data, there is nothing
>>>>>>>>> as an answer which is supposed to be late and correct. The timeliness is
>>>>>>>>> part of the application. if I get the right answer too slowly it becomes
>>>>>>>>> useless or wrong" is actually fundamental to *why* we need this
>>>>>>>>> Spark Structured Streaming proposal.
>>>>>>>>>
>>>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>>>> applications where, as I define it, the *timeliness* of the
>>>>>>>>> answer is as critical as its *correctness*. Spark's current
>>>>>>>>> streaming engine, primarily operating on micro-batches, often delivers
>>>>>>>>> results that are technically "correct" but arrive too late to be truly
>>>>>>>>> useful for certain high-stakes, real-time scenarios. This makes them
>>>>>>>>> "useless or wrong" in a practical, business-critical sense.
>>>>>>>>>
>>>>>>>>> For example *in real-time fraud detection* and In *high-frequency
>>>>>>>>> trading,* market data or trade execution commands must be
>>>>>>>>> delivered with minimal latency. Even a slight delay can mean missed
>>>>>>>>> opportunities or significant financial losses, making a "correct" price
>>>>>>>>> update useless if it's not instantaneous. able for these
>>>>>>>>> demanding use cases, where a "late but correct" answer is simply not good
>>>>>>>>> enough. As a colliery it is a fundamental concept, so it has to be treated
>>>>>>>>> as such not as a comment.in SPIP
>>>>>>>>>
>>>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>> GDPR
>>>>>>>>>
>>>>>>>>> view my Linkedin profile
>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hey Mich,
>>>>>>>>>>
>>>>>>>>>> Sorry, I may be missing something here but what does your
>>>>>>>>>> definition here have to do with the SPIP? Perhaps add comments directly
>>>>>>>>>> to the SPIP to provide context as the code snippet below is a direct copy
>>>>>>>>>> from the SPIP itself.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Denny
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> just to add
>>>>>>>>>>>
>>>>>>>>>>> A stronger definition of real time. The engineering definition
>>>>>>>>>>> of real time is roughly fast enough to be interactive
>>>>>>>>>>>
>>>>>>>>>>> However, I put a stronger definition. In real time application
>>>>>>>>>>> or data, there is nothing as an answer which is supposed to be late and
>>>>>>>>>>> correct. The timeliness is part of the application.if I get the right
>>>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>>>> GDPR
>>>>>>>>>>>
>>>>>>>>>>> view my Linkedin profile
>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The current limitations in SSS come from micro-batching.If you
>>>>>>>>>>>> are going to reduce micro-batching, this reduction must be balanced against
>>>>>>>>>>>> the available processing capacity of the cluster to prevent back pressure
>>>>>>>>>>>> and instability. In the case of Continuous Processing mode, a
>>>>>>>>>>>> specific continuous trigger with a desired checkpoint interval quote
>>>>>>>>>>>>
>>>>>>>>>>>> "
>>>>>>>>>>>> df.writeStream
>>>>>>>>>>>> .format("...")
>>>>>>>>>>>> .option("...")
>>>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger
>>>>>>>>>>>> type to enable real-time Mode
>>>>>>>>>>>> .start()
>>>>>>>>>>>> This Trigger.RealTime signals that the query should run in the
>>>>>>>>>>>> new ultra low-latency execution mode. A time interval can also be
>>>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each micro-batch should
>>>>>>>>>>>> run for.
>>>>>>>>>>>> "
>>>>>>>>>>>>
>>>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>>>>>> HTH
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis
>>>>>>>>>>>> | GDPR
>>>>>>>>>>>>
>>>>>>>>>>>> view my Linkedin profile
>>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that I've been
>>>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and
>>>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>>>>>>>> ].
>>>>>>>>>>>>>
>>>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode”
>>>>>>>>>>>>> in Spark Structured Streaming that significantly lowers end-to-end latency
>>>>>>>>>>>>> for processing streams of data.
>>>>>>>>>>>>>
>>>>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is
>>>>>>>>>>>>> to make Spark capable of handling streaming jobs that need results almost
>>>>>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>>>>>>>> engine.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>>>>>>>> suggestions on this approach!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best,
>>>>>>> Yanbo
>>>>>>>
>>>>>>
>>>
>>> --
>>> John Zhuge
>>>
>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Xiao Li <ga...@gmail.com>.
+1
Yuming Wang <yu...@apache.org> 于2025年5月29日周四 02:22写道：
> +1.
>
> On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
>
>> +1
>> Sent from my iPhone
>>
>> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
>>
>>
>> +1 Nice feature
>>
>> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xy...@gmail.com>
>> wrote:
>>
>>> +1
>>>
>>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>>
>>>> +1, LGTM.
>>>>
>>>> Kent
>>>>
>>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>>
>>>>> +1. Super excited by this initiative!
>>>>>
>>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>>>>>> eliminate the need for separate streaming engines, reducing system
>>>>>>> complexity and operational cost. Excited to see this direction!
>>>>>>>
>>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> My point about "in real time application or data, there is nothing
>>>>>>>> as an answer which is supposed to be late and correct. The timeliness is
>>>>>>>> part of the application. if I get the right answer too slowly it becomes
>>>>>>>> useless or wrong" is actually fundamental to *why* we need this
>>>>>>>> Spark Structured Streaming proposal.
>>>>>>>>
>>>>>>>> The proposal is precisely about enabling Spark to power
>>>>>>>> applications where, as I define it, the *timeliness* of the answer
>>>>>>>> is as critical as its *correctness*. Spark's current streaming
>>>>>>>> engine, primarily operating on micro-batches, often delivers results that
>>>>>>>> are technically "correct" but arrive too late to be truly useful for
>>>>>>>> certain high-stakes, real-time scenarios. This makes them "useless or
>>>>>>>> wrong" in a practical, business-critical sense.
>>>>>>>>
>>>>>>>> For example *in real-time fraud detection* and In *high-frequency
>>>>>>>> trading,* market data or trade execution commands must be
>>>>>>>> delivered with minimal latency. Even a slight delay can mean missed
>>>>>>>> opportunities or significant financial losses, making a "correct" price
>>>>>>>> update useless if it's not instantaneous. able for these demanding
>>>>>>>> use cases, where a "late but correct" answer is simply not good enough. As
>>>>>>>> a colliery it is a fundamental concept, so it has to be treated as such not
>>>>>>>> as a comment.in SPIP
>>>>>>>>
>>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>> GDPR
>>>>>>>>
>>>>>>>> view my Linkedin profile
>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hey Mich,
>>>>>>>>>
>>>>>>>>> Sorry, I may be missing something here but what does your
>>>>>>>>> definition here have to do with the SPIP? Perhaps add comments directly
>>>>>>>>> to the SPIP to provide context as the code snippet below is a direct copy
>>>>>>>>> from the SPIP itself.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Denny
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> just to add
>>>>>>>>>>
>>>>>>>>>> A stronger definition of real time. The engineering definition of
>>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>>>
>>>>>>>>>> However, I put a stronger definition. In real time application or
>>>>>>>>>> data, there is nothing as an answer which is supposed to be late and
>>>>>>>>>> correct. The timeliness is part of the application.if I get the right
>>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>>> GDPR
>>>>>>>>>>
>>>>>>>>>> view my Linkedin profile
>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> The current limitations in SSS come from micro-batching.If you
>>>>>>>>>>> are going to reduce micro-batching, this reduction must be balanced against
>>>>>>>>>>> the available processing capacity of the cluster to prevent back pressure
>>>>>>>>>>> and instability. In the case of Continuous Processing mode, a
>>>>>>>>>>> specific continuous trigger with a desired checkpoint interval quote
>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>> df.writeStream
>>>>>>>>>>> .format("...")
>>>>>>>>>>> .option("...")
>>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger
>>>>>>>>>>> type to enable real-time Mode
>>>>>>>>>>> .start()
>>>>>>>>>>> This Trigger.RealTime signals that the query should run in the
>>>>>>>>>>> new ultra low-latency execution mode. A time interval can also be
>>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each micro-batch should
>>>>>>>>>>> run for.
>>>>>>>>>>> "
>>>>>>>>>>>
>>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>>>>> HTH
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>>>> GDPR
>>>>>>>>>>>
>>>>>>>>>>> view my Linkedin profile
>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that I've been
>>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and
>>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>>>>>>> ].
>>>>>>>>>>>>
>>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode”
>>>>>>>>>>>> in Spark Structured Streaming that significantly lowers end-to-end latency
>>>>>>>>>>>> for processing streams of data.
>>>>>>>>>>>>
>>>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is
>>>>>>>>>>>> to make Spark capable of handling streaming jobs that need results almost
>>>>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>>>>>
>>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>>>>>>> engine.
>>>>>>>>>>>>
>>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and
>>>>>>>>>>>> suggestions on this approach!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best,
>>>>>> Yanbo
>>>>>>
>>>>>
>>
>> --
>> John Zhuge
>>
>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Yuming Wang <yu...@apache.org>.
+1.
On Thu, May 29, 2025 at 3:36 PM DB Tsai <db...@dbtsai.com> wrote:
> +1
> Sent from my iPhone
>
> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
>
>
> +1 Nice feature
>
> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xy...@gmail.com>
> wrote:
>
>> +1
>>
>> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>>
>>> +1, LGTM.
>>>
>>> Kent
>>>
>>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>>
>>>> +1. Super excited by this initiative!
>>>>
>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>> By unifying batch and low-latency streaming in Spark, we can
>>>>>> eliminate the need for separate streaming engines, reducing system
>>>>>> complexity and operational cost. Excited to see this direction!
>>>>>>
>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> My point about "in real time application or data, there is nothing
>>>>>>> as an answer which is supposed to be late and correct. The timeliness is
>>>>>>> part of the application. if I get the right answer too slowly it becomes
>>>>>>> useless or wrong" is actually fundamental to *why* we need this
>>>>>>> Spark Structured Streaming proposal.
>>>>>>>
>>>>>>> The proposal is precisely about enabling Spark to power applications
>>>>>>> where, as I define it, the *timeliness* of the answer is as
>>>>>>> critical as its *correctness*. Spark's current streaming engine,
>>>>>>> primarily operating on micro-batches, often delivers results that are
>>>>>>> technically "correct" but arrive too late to be truly useful for certain
>>>>>>> high-stakes, real-time scenarios. This makes them "useless or wrong" in a
>>>>>>> practical, business-critical sense.
>>>>>>>
>>>>>>> For example *in real-time fraud detection* and In *high-frequency
>>>>>>> trading,* market data or trade execution commands must be delivered
>>>>>>> with minimal latency. Even a slight delay can mean missed opportunities or
>>>>>>> significant financial losses, making a "correct" price update useless if
>>>>>>> it's not instantaneous. able for these demanding use cases, where a
>>>>>>> "late but correct" answer is simply not good enough. As a colliery it is a
>>>>>>> fundamental concept, so it has to be treated as such not as a
>>>>>>> comment.in SPIP
>>>>>>>
>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>> Dr Mich Talebzadeh,
>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Mich,
>>>>>>>>
>>>>>>>> Sorry, I may be missing something here but what does your
>>>>>>>> definition here have to do with the SPIP? Perhaps add comments directly
>>>>>>>> to the SPIP to provide context as the code snippet below is a direct copy
>>>>>>>> from the SPIP itself.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Denny
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> just to add
>>>>>>>>>
>>>>>>>>> A stronger definition of real time. The engineering definition of
>>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>>
>>>>>>>>> However, I put a stronger definition. In real time application or
>>>>>>>>> data, there is nothing as an answer which is supposed to be late and
>>>>>>>>> correct. The timeliness is part of the application.if I get the right
>>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>> GDPR
>>>>>>>>>
>>>>>>>>> view my Linkedin profile
>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> The current limitations in SSS come from micro-batching.If you
>>>>>>>>>> are going to reduce micro-batching, this reduction must be balanced against
>>>>>>>>>> the available processing capacity of the cluster to prevent back pressure
>>>>>>>>>> and instability. In the case of Continuous Processing mode, a
>>>>>>>>>> specific continuous trigger with a desired checkpoint interval quote
>>>>>>>>>>
>>>>>>>>>> "
>>>>>>>>>> df.writeStream
>>>>>>>>>> .format("...")
>>>>>>>>>> .option("...")
>>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger
>>>>>>>>>> type to enable real-time Mode
>>>>>>>>>> .start()
>>>>>>>>>> This Trigger.RealTime signals that the query should run in the
>>>>>>>>>> new ultra low-latency execution mode. A time interval can also be
>>>>>>>>>> specified, e.g. “300 Seconds”, to indicate how long each micro-batch should
>>>>>>>>>> run for.
>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>>>> HTH
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>>> GDPR
>>>>>>>>>>
>>>>>>>>>> view my Linkedin profile
>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that I've been
>>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and
>>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>>>>>> ].
>>>>>>>>>>>
>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode”
>>>>>>>>>>> in Spark Structured Streaming that significantly lowers end-to-end latency
>>>>>>>>>>> for processing streams of data.
>>>>>>>>>>>
>>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is
>>>>>>>>>>> to make Spark capable of handling streaming jobs that need results almost
>>>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>>>>
>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>>>>>> engine.
>>>>>>>>>>>
>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions
>>>>>>>>>>> on this approach!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>
>>>>> --
>>>>> Best,
>>>>> Yanbo
>>>>>
>>>>
>
> --
> John Zhuge
>
>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by DB Tsai <db...@dbtsai.com>.
+1
Sent from my iPhone
> On May 29, 2025, at 12:15 AM, John Zhuge <jz...@apache.org> wrote:
>
>
>
>
> +1 Nice feature
>
>
>
>
>
> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li
> <[xyliyuanjian@gmail.com](mailto:xyliyuanjian@gmail.com)> wrote:
>
>
>> +1
>
>>
>>
>
>>
>> Kent Yao <[yao@apache.org](mailto:yao@apache.org)> 于2025年5月28日周三 19:31写道：
>
>>
>>> +1, LGTM.
>>>
>>>
>
>>>
>>> Kent
>>>
>>>
> 在 2025年5月29日星期四，Chao Sun <[sunchao@apache.org](mailto:sunchao@apache.org)>
> 写道：
>
>>>
>>>> +1. Super excited by this initiative!
>
>>>>
>>>>
>
>>>>
>>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang
<[ybliang8@gmail.com](mailto:ybliang8@gmail.com)> wrote:
>
>>>>
>>>>> +1
>
>>>>>
>>>>>
>
>>>>>
>>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao
<[huaxin.gao11@gmail.com](mailto:huaxin.gao11@gmail.com)> wrote:
>
>>>>>
>>>>>> +1
>>>>>>
>>>>>> By unifying batch and low-latency streaming in Spark, we can eliminate
the need for separate streaming engines, reducing system complexity and
operational cost. Excited to see this direction!
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh
<[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:
>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> My point about "in real time application or data, there is nothing as
an answer which is supposed to be late and correct. The timeliness is part of
the application. if I get the right answer too slowly it becomes useless or
wrong" is actually fundamental to _why_ we need this Spark Structured
Streaming proposal.
>>>>>>>
>>>>>>> The proposal is precisely about enabling Spark to power applications
where, as I define it, the _timeliness_ of the answer is as critical as its
_correctness_. Spark's current streaming engine, primarily operating on micro-
batches, often delivers results that are technically "correct" but arrive too
late to be truly useful for certain high-stakes, real-time scenarios. This
makes them "useless or wrong" in a practical, business-critical sense.
>>>>>>>
>>>>>>> For example _in real-time fraud detection_ and In _high-frequency
trading,_ market data or trade execution commands must be delivered with
minimal latency. Even a slight delay can mean missed opportunities or
significant financial losses, making a "correct" price update useless if it's
not instantaneous. able for these demanding use cases, where a "late but
correct" answer is simply not good enough. As a colliery it is a fundamental
concept, so it has to be treated as such not as a
[comment.in](http://comment.in) SPIP
>>>>>>>
>>>>>>> Hope this clarifies the connection in practical terms
>>>>>>>
>>>>>>> Dr Mich Talebzadeh,
>>>>>>>
>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> 
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee
<[denny.g.lee@gmail.com](mailto:denny.g.lee@gmail.com)> wrote:
>
>>>>>>>
>>>>>>>> Hey Mich,
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> Sorry, I may be missing something here but what does your definition
here have to do with the SPIP? Perhaps add comments directly to the SPIP to
provide context as the code snippet below is a direct copy from the SPIP
itself.
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Denny
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>>
>
>>>>>>>>
>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh
<[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:
>
>>>>>>>>
>>>>>>>>> just to add
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> A stronger definition of real time. The engineering definition of
real time is roughly fast enough to be interactive
>>>>>>>>>
>>>>>>>>> However, I put a stronger definition. In real time application or
data, there is nothing as an answer which is supposed to be late and correct.
The timeliness is part of the application.if I get the right answer too slowly
it becomes useless or wrong
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>
>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
GDPR
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> 
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
>>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh
<[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:
>
>>>>>>>>>
>>>>>>>>>> The current limitations in SSS come from micro-batching.If you are
going to reduce micro-batching, this reduction must be balanced against the
available processing capacity of the cluster to prevent back pressure and
instability. In the case of Continuous Processing mode, a specific continuous
trigger with a desired checkpoint interval quote
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>> df.writeStream
> .format("...")
> .option("...")
> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type to
> enable real-time Mode
> .start()
> This Trigger.RealTime signals that the query should run in the new ultra
> low-latency execution mode. A time interval can also be specified, e.g.
> “300 Seconds”, to indicate how long each micro-batch should run for.
>>>>>>>>>>
>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>>>>
>>>>>>>>>> HTH
>>>>>>>>>>
>>>>>>>>>>
>
>
>>>>>>>>>>
>>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>>>
>>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
GDPR
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> 
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>>
>>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng
<[jerry.boyang.peng@gmail.com](mailto:jerry.boyang.peng@gmail.com)> wrote:
>
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>>>>>>>
>>>>>>>>>>> I want to start a discussion thread for the SPIP titled “Real-Time
Mode in Apache Spark Structured Streaming” that I've been working on with
Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust:
[[JIRA](https://issues.apache.org/jira/browse/SPARK-52330)]
[[Doc](https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing)].
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>>>>>>>
>>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
Spark Structured Streaming that significantly lowers end-to-end latency for
processing streams of data.
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>>>>>>>
>>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is to
make Spark capable of handling streaming jobs that need results almost
immediately (within O(100) milliseconds). We want to achieve this without
changing the high-level DataFrame/Dataset API that users already use – so
existing streaming queries can run in this new ultra-low-latency mode by
simply turning it on, without rewriting their logic.
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>>>>>>>
>>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
applications (like instant anomaly alerts or live personalization) that today
cannot meet their latency requirements with Spark’s current streaming engine.
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>>>>>>>
>>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions
on this approach!
>>>>>>>>>>>
>>>>>>>>>>>
>
>>>>>
>>>>>
>
>>>>>
>>>>>
>
>>>>>
>>>>> \--
>
>>>>>
>>>>> Best,
>>>>>
>>>>> Yanbo
>
>
>
>
>
>
>
> \--
>
>
> John Zhuge
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by John Zhuge <jz...@apache.org>.
+1 Nice feature
On Wed, May 28, 2025 at 9:53 PM Yuanjian Li <xy...@gmail.com> wrote:
> +1
>
> Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
>
>> +1, LGTM.
>>
>> Kent
>>
>> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>>
>>> +1. Super excited by this initiative!
>>>
>>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1
>>>>> By unifying batch and low-latency streaming in Spark, we can eliminate
>>>>> the need for separate streaming engines, reducing system complexity and
>>>>> operational cost. Excited to see this direction!
>>>>>
>>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> My point about "in real time application or data, there is nothing as
>>>>>> an answer which is supposed to be late and correct. The timeliness is part
>>>>>> of the application. if I get the right answer too slowly it becomes useless
>>>>>> or wrong" is actually fundamental to *why* we need this Spark
>>>>>> Structured Streaming proposal.
>>>>>>
>>>>>> The proposal is precisely about enabling Spark to power applications
>>>>>> where, as I define it, the *timeliness* of the answer is as critical
>>>>>> as its *correctness*. Spark's current streaming engine, primarily
>>>>>> operating on micro-batches, often delivers results that are technically
>>>>>> "correct" but arrive too late to be truly useful for certain high-stakes,
>>>>>> real-time scenarios. This makes them "useless or wrong" in a practical,
>>>>>> business-critical sense.
>>>>>>
>>>>>> For example *in real-time fraud detection* and In *high-frequency
>>>>>> trading,* market data or trade execution commands must be delivered
>>>>>> with minimal latency. Even a slight delay can mean missed opportunities or
>>>>>> significant financial losses, making a "correct" price update useless if
>>>>>> it's not instantaneous. able for these demanding use cases, where a
>>>>>> "late but correct" answer is simply not good enough. As a colliery it is a
>>>>>> fundamental concept, so it has to be treated as such not as a
>>>>>> comment.in SPIP
>>>>>>
>>>>>> Hope this clarifies the connection in practical terms
>>>>>> Dr Mich Talebzadeh,
>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey Mich,
>>>>>>>
>>>>>>> Sorry, I may be missing something here but what does your definition
>>>>>>> here have to do with the SPIP? Perhaps add comments directly to the SPIP
>>>>>>> to provide context as the code snippet below is a direct copy from the SPIP
>>>>>>> itself.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Denny
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>
>>>>>>>> just to add
>>>>>>>>
>>>>>>>> A stronger definition of real time. The engineering definition of
>>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>>
>>>>>>>> However, I put a stronger definition. In real time application or
>>>>>>>> data, there is nothing as an answer which is supposed to be late and
>>>>>>>> correct. The timeliness is part of the application.if I get the right
>>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>> GDPR
>>>>>>>>
>>>>>>>> view my Linkedin profile
>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> The current limitations in SSS come from micro-batching.If you are
>>>>>>>>> going to reduce micro-batching, this reduction must be balanced against the
>>>>>>>>> available processing capacity of the cluster to prevent back pressure and
>>>>>>>>> instability. In the case of Continuous Processing mode, a
>>>>>>>>> specific continuous trigger with a desired checkpoint interval quote
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> df.writeStream
>>>>>>>>> .format("...")
>>>>>>>>> .option("...")
>>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger
>>>>>>>>> type to enable real-time Mode
>>>>>>>>> .start()
>>>>>>>>> This Trigger.RealTime signals that the query should run in the new
>>>>>>>>> ultra low-latency execution mode. A time interval can also be specified,
>>>>>>>>> e.g. “300 Seconds”, to indicate how long each micro-batch should run for.
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>>> HTH
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>>> GDPR
>>>>>>>>>
>>>>>>>>> view my Linkedin profile
>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I want to start a discussion thread for the SPIP titled
>>>>>>>>>> “Real-Time Mode in Apache Spark Structured Streaming” that I've been
>>>>>>>>>> working on with Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and
>>>>>>>>>> Michael Armbrust: [JIRA
>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>>>>> ].
>>>>>>>>>>
>>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
>>>>>>>>>> Spark Structured Streaming that significantly lowers end-to-end latency for
>>>>>>>>>> processing streams of data.
>>>>>>>>>>
>>>>>>>>>> A key principle of this proposal is compatibility. Our goal is to
>>>>>>>>>> make Spark capable of handling streaming jobs that need results almost
>>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>>>
>>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>>>>> engine.
>>>>>>>>>>
>>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions
>>>>>>>>>> on this approach!
>>>>>>>>>>
>>>>>>>>>>
>>>>
>>>> --
>>>> Best,
>>>> Yanbo
>>>>
>>>
--
John Zhuge
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Yuanjian Li <xy...@gmail.com>.
+1
Kent Yao <ya...@apache.org> 于2025年5月28日周三 19:31写道：
> +1, LGTM.
>
> Kent
>
> 在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
>
>> +1. Super excited by this initiative!
>>
>> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
>>> wrote:
>>>
>>>> +1
>>>> By unifying batch and low-latency streaming in Spark, we can eliminate
>>>> the need for separate streaming engines, reducing system complexity and
>>>> operational cost. Excited to see this direction!
>>>>
>>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> My point about "in real time application or data, there is nothing as
>>>>> an answer which is supposed to be late and correct. The timeliness is part
>>>>> of the application. if I get the right answer too slowly it becomes useless
>>>>> or wrong" is actually fundamental to *why* we need this Spark
>>>>> Structured Streaming proposal.
>>>>>
>>>>> The proposal is precisely about enabling Spark to power applications
>>>>> where, as I define it, the *timeliness* of the answer is as critical
>>>>> as its *correctness*. Spark's current streaming engine, primarily
>>>>> operating on micro-batches, often delivers results that are technically
>>>>> "correct" but arrive too late to be truly useful for certain high-stakes,
>>>>> real-time scenarios. This makes them "useless or wrong" in a practical,
>>>>> business-critical sense.
>>>>>
>>>>> For example *in real-time fraud detection* and In *high-frequency
>>>>> trading,* market data or trade execution commands must be delivered
>>>>> with minimal latency. Even a slight delay can mean missed opportunities or
>>>>> significant financial losses, making a "correct" price update useless if
>>>>> it's not instantaneous. able for these demanding use cases, where a
>>>>> "late but correct" answer is simply not good enough. As a colliery it is a
>>>>> fundamental concept, so it has to be treated as such not as a
>>>>> comment.in SPIP
>>>>>
>>>>> Hope this clarifies the connection in practical terms
>>>>> Dr Mich Talebzadeh,
>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com> wrote:
>>>>>
>>>>>> Hey Mich,
>>>>>>
>>>>>> Sorry, I may be missing something here but what does your definition
>>>>>> here have to do with the SPIP? Perhaps add comments directly to the SPIP
>>>>>> to provide context as the code snippet below is a direct copy from the SPIP
>>>>>> itself.
>>>>>>
>>>>>> Thanks,
>>>>>> Denny
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> just to add
>>>>>>>
>>>>>>> A stronger definition of real time. The engineering definition of
>>>>>>> real time is roughly fast enough to be interactive
>>>>>>>
>>>>>>> However, I put a stronger definition. In real time application or
>>>>>>> data, there is nothing as an answer which is supposed to be late and
>>>>>>> correct. The timeliness is part of the application.if I get the right
>>>>>>> answer too slowly it becomes useless or wrong
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dr Mich Talebzadeh,
>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>
>>>>>>>> The current limitations in SSS come from micro-batching.If you are
>>>>>>>> going to reduce micro-batching, this reduction must be balanced against the
>>>>>>>> available processing capacity of the cluster to prevent back pressure and
>>>>>>>> instability. In the case of Continuous Processing mode, a specific
>>>>>>>> continuous trigger with a desired checkpoint interval quote
>>>>>>>>
>>>>>>>> "
>>>>>>>> df.writeStream
>>>>>>>> .format("...")
>>>>>>>> .option("...")
>>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type
>>>>>>>> to enable real-time Mode
>>>>>>>> .start()
>>>>>>>> This Trigger.RealTime signals that the query should run in the new
>>>>>>>> ultra low-latency execution mode. A time interval can also be specified,
>>>>>>>> e.g. “300 Seconds”, to indicate how long each micro-batch should run for.
>>>>>>>> "
>>>>>>>>
>>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>>> HTH
>>>>>>>>
>>>>>>>>
>>>>>>>> Dr Mich Talebzadeh,
>>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis |
>>>>>>>> GDPR
>>>>>>>>
>>>>>>>> view my Linkedin profile
>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I want to start a discussion thread for the SPIP titled “Real-Time
>>>>>>>>> Mode in Apache Spark Structured Streaming” that I've been working on with
>>>>>>>>> Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [
>>>>>>>>> JIRA <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>>>> ].
>>>>>>>>>
>>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
>>>>>>>>> Spark Structured Streaming that significantly lowers end-to-end latency for
>>>>>>>>> processing streams of data.
>>>>>>>>>
>>>>>>>>> A key principle of this proposal is compatibility. Our goal is to
>>>>>>>>> make Spark capable of handling streaming jobs that need results almost
>>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>>
>>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>>>> engine.
>>>>>>>>>
>>>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions
>>>>>>>>> on this approach!
>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Best,
>>> Yanbo
>>>
>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Kent Yao <ya...@apache.org>.
+1, LGTM.
Kent
在 2025年5月29日星期四，Chao Sun <su...@apache.org> 写道：
> +1. Super excited by this initiative!
>
> On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com> wrote:
>
>> +1
>>
>> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
>> wrote:
>>
>>> +1
>>> By unifying batch and low-latency streaming in Spark, we can eliminate
>>> the need for separate streaming engines, reducing system complexity and
>>> operational cost. Excited to see this direction!
>>>
>>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> My point about "in real time application or data, there is nothing as
>>>> an answer which is supposed to be late and correct. The timeliness is part
>>>> of the application. if I get the right answer too slowly it becomes useless
>>>> or wrong" is actually fundamental to *why* we need this Spark
>>>> Structured Streaming proposal.
>>>>
>>>> The proposal is precisely about enabling Spark to power applications
>>>> where, as I define it, the *timeliness* of the answer is as critical
>>>> as its *correctness*. Spark's current streaming engine, primarily
>>>> operating on micro-batches, often delivers results that are technically
>>>> "correct" but arrive too late to be truly useful for certain high-stakes,
>>>> real-time scenarios. This makes them "useless or wrong" in a practical,
>>>> business-critical sense.
>>>>
>>>> For example *in real-time fraud detection* and In *high-frequency
>>>> trading,* market data or trade execution commands must be delivered
>>>> with minimal latency. Even a slight delay can mean missed opportunities or
>>>> significant financial losses, making a "correct" price update useless if
>>>> it's not instantaneous. able for these demanding use cases, where a
>>>> "late but correct" answer is simply not good enough. As a colliery it is a
>>>> fundamental concept, so it has to be treated as such not as a
>>>> comment.in SPIP
>>>>
>>>> Hope this clarifies the connection in practical terms
>>>> Dr Mich Talebzadeh,
>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com> wrote:
>>>>
>>>>> Hey Mich,
>>>>>
>>>>> Sorry, I may be missing something here but what does your definition
>>>>> here have to do with the SPIP? Perhaps add comments directly to the SPIP
>>>>> to provide context as the code snippet below is a direct copy from the SPIP
>>>>> itself.
>>>>>
>>>>> Thanks,
>>>>> Denny
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> just to add
>>>>>>
>>>>>> A stronger definition of real time. The engineering definition of
>>>>>> real time is roughly fast enough to be interactive
>>>>>>
>>>>>> However, I put a stronger definition. In real time application or
>>>>>> data, there is nothing as an answer which is supposed to be late and
>>>>>> correct. The timeliness is part of the application.if I get the right
>>>>>> answer too slowly it becomes useless or wrong
>>>>>>
>>>>>>
>>>>>>
>>>>>> Dr Mich Talebzadeh,
>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> The current limitations in SSS come from micro-batching.If you are
>>>>>>> going to reduce micro-batching, this reduction must be balanced against the
>>>>>>> available processing capacity of the cluster to prevent back pressure and
>>>>>>> instability. In the case of Continuous Processing mode, a specific
>>>>>>> continuous trigger with a desired checkpoint interval quote
>>>>>>>
>>>>>>> "
>>>>>>> df.writeStream
>>>>>>> .format("...")
>>>>>>> .option("...")
>>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type
>>>>>>> to enable real-time Mode
>>>>>>> .start()
>>>>>>> This Trigger.RealTime signals that the query should run in the new
>>>>>>> ultra low-latency execution mode. A time interval can also be specified,
>>>>>>> e.g. “300 Seconds”, to indicate how long each micro-batch should run for.
>>>>>>> "
>>>>>>>
>>>>>>> will inevitably depend on many factors. Not that simple
>>>>>>> HTH
>>>>>>>
>>>>>>>
>>>>>>> Dr Mich Talebzadeh,
>>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <
>>>>>>> jerry.boyang.peng@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I want to start a discussion thread for the SPIP titled “Real-Time
>>>>>>>> Mode in Apache Spark Structured Streaming” that I've been working on with
>>>>>>>> Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [
>>>>>>>> JIRA <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>>> ].
>>>>>>>>
>>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
>>>>>>>> Spark Structured Streaming that significantly lowers end-to-end latency for
>>>>>>>> processing streams of data.
>>>>>>>>
>>>>>>>> A key principle of this proposal is compatibility. Our goal is to
>>>>>>>> make Spark capable of handling streaming jobs that need results almost
>>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>>
>>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>>> engine.
>>>>>>>>
>>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions on
>>>>>>>> this approach!
>>>>>>>>
>>>>>>>>
>>
>> --
>> Best,
>> Yanbo
>>
>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Chao Sun <su...@apache.org>.
+1. Super excited by this initiative!
On Wed, May 28, 2025 at 1:54 PM Yanbo Liang <yb...@gmail.com> wrote:
> +1
>
> On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com>
> wrote:
>
>> +1
>> By unifying batch and low-latency streaming in Spark, we can eliminate
>> the need for separate streaming engines, reducing system complexity and
>> operational cost. Excited to see this direction!
>>
>> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> My point about "in real time application or data, there is nothing as an
>>> answer which is supposed to be late and correct. The timeliness is part of
>>> the application. if I get the right answer too slowly it becomes useless or
>>> wrong" is actually fundamental to *why* we need this Spark Structured
>>> Streaming proposal.
>>>
>>> The proposal is precisely about enabling Spark to power applications
>>> where, as I define it, the *timeliness* of the answer is as critical as
>>> its *correctness*. Spark's current streaming engine, primarily
>>> operating on micro-batches, often delivers results that are technically
>>> "correct" but arrive too late to be truly useful for certain high-stakes,
>>> real-time scenarios. This makes them "useless or wrong" in a practical,
>>> business-critical sense.
>>>
>>> For example *in real-time fraud detection* and In *high-frequency
>>> trading,* market data or trade execution commands must be delivered
>>> with minimal latency. Even a slight delay can mean missed opportunities or
>>> significant financial losses, making a "correct" price update useless if
>>> it's not instantaneous. able for these demanding use cases, where a
>>> "late but correct" answer is simply not good enough. As a colliery it is a
>>> fundamental concept, so it has to be treated as such not as a comment.in
>>> SPIP
>>>
>>> Hope this clarifies the connection in practical terms
>>> Dr Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com> wrote:
>>>
>>>> Hey Mich,
>>>>
>>>> Sorry, I may be missing something here but what does your definition
>>>> here have to do with the SPIP? Perhaps add comments directly to the SPIP
>>>> to provide context as the code snippet below is a direct copy from the SPIP
>>>> itself.
>>>>
>>>> Thanks,
>>>> Denny
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> just to add
>>>>>
>>>>> A stronger definition of real time. The engineering definition of real
>>>>> time is roughly fast enough to be interactive
>>>>>
>>>>> However, I put a stronger definition. In real time application or
>>>>> data, there is nothing as an answer which is supposed to be late and
>>>>> correct. The timeliness is part of the application.if I get the right
>>>>> answer too slowly it becomes useless or wrong
>>>>>
>>>>>
>>>>>
>>>>> Dr Mich Talebzadeh,
>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> The current limitations in SSS come from micro-batching.If you are
>>>>>> going to reduce micro-batching, this reduction must be balanced against the
>>>>>> available processing capacity of the cluster to prevent back pressure and
>>>>>> instability. In the case of Continuous Processing mode, a specific
>>>>>> continuous trigger with a desired checkpoint interval quote
>>>>>>
>>>>>> "
>>>>>> df.writeStream
>>>>>> .format("...")
>>>>>> .option("...")
>>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type
>>>>>> to enable real-time Mode
>>>>>> .start()
>>>>>> This Trigger.RealTime signals that the query should run in the new
>>>>>> ultra low-latency execution mode. A time interval can also be specified,
>>>>>> e.g. “300 Seconds”, to indicate how long each micro-batch should run for.
>>>>>> "
>>>>>>
>>>>>> will inevitably depend on many factors. Not that simple
>>>>>> HTH
>>>>>>
>>>>>>
>>>>>> Dr Mich Talebzadeh,
>>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <je...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I want to start a discussion thread for the SPIP titled “Real-Time
>>>>>>> Mode in Apache Spark Structured Streaming” that I've been working on with
>>>>>>> Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [
>>>>>>> JIRA <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>>> ].
>>>>>>>
>>>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
>>>>>>> Spark Structured Streaming that significantly lowers end-to-end latency for
>>>>>>> processing streams of data.
>>>>>>>
>>>>>>> A key principle of this proposal is compatibility. Our goal is to
>>>>>>> make Spark capable of handling streaming jobs that need results almost
>>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>>> simply turning it on, without rewriting their logic.
>>>>>>>
>>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>>> engine.
>>>>>>>
>>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions on
>>>>>>> this approach!
>>>>>>>
>>>>>>>
>
> --
> Best,
> Yanbo
>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Yanbo Liang <yb...@gmail.com>.
+1
On Wed, May 28, 2025 at 12:34 PM huaxin gao <hu...@gmail.com> wrote:
> +1
> By unifying batch and low-latency streaming in Spark, we can eliminate the
> need for separate streaming engines, reducing system complexity and
> operational cost. Excited to see this direction!
>
> On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> My point about "in real time application or data, there is nothing as an
>> answer which is supposed to be late and correct. The timeliness is part of
>> the application. if I get the right answer too slowly it becomes useless or
>> wrong" is actually fundamental to *why* we need this Spark Structured
>> Streaming proposal.
>>
>> The proposal is precisely about enabling Spark to power applications
>> where, as I define it, the *timeliness* of the answer is as critical as
>> its *correctness*. Spark's current streaming engine, primarily operating
>> on micro-batches, often delivers results that are technically "correct" but
>> arrive too late to be truly useful for certain high-stakes, real-time
>> scenarios. This makes them "useless or wrong" in a practical,
>> business-critical sense.
>>
>> For example *in real-time fraud detection* and In *high-frequency
>> trading,* market data or trade execution commands must be delivered with
>> minimal latency. Even a slight delay can mean missed opportunities or
>> significant financial losses, making a "correct" price update useless if
>> it's not instantaneous. able for these demanding use cases, where a
>> "late but correct" answer is simply not good enough. As a colliery it is a
>> fundamental concept, so it has to be treated as such not as a comment.in
>> SPIP
>>
>> Hope this clarifies the connection in practical terms
>> Dr Mich Talebzadeh,
>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com> wrote:
>>
>>> Hey Mich,
>>>
>>> Sorry, I may be missing something here but what does your definition
>>> here have to do with the SPIP? Perhaps add comments directly to the SPIP
>>> to provide context as the code snippet below is a direct copy from the SPIP
>>> itself.
>>>
>>> Thanks,
>>> Denny
>>>
>>>
>>>
>>>
>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>>> just to add
>>>>
>>>> A stronger definition of real time. The engineering definition of real
>>>> time is roughly fast enough to be interactive
>>>>
>>>> However, I put a stronger definition. In real time application or data,
>>>> there is nothing as an answer which is supposed to be late and correct. The
>>>> timeliness is part of the application.if I get the right answer too slowly
>>>> it becomes useless or wrong
>>>>
>>>>
>>>>
>>>> Dr Mich Talebzadeh,
>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> The current limitations in SSS come from micro-batching.If you are
>>>>> going to reduce micro-batching, this reduction must be balanced against the
>>>>> available processing capacity of the cluster to prevent back pressure and
>>>>> instability. In the case of Continuous Processing mode, a specific
>>>>> continuous trigger with a desired checkpoint interval quote
>>>>>
>>>>> "
>>>>> df.writeStream
>>>>> .format("...")
>>>>> .option("...")
>>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type to
>>>>> enable real-time Mode
>>>>> .start()
>>>>> This Trigger.RealTime signals that the query should run in the new
>>>>> ultra low-latency execution mode. A time interval can also be specified,
>>>>> e.g. “300 Seconds”, to indicate how long each micro-batch should run for.
>>>>> "
>>>>>
>>>>> will inevitably depend on many factors. Not that simple
>>>>> HTH
>>>>>
>>>>>
>>>>> Dr Mich Talebzadeh,
>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <je...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I want to start a discussion thread for the SPIP titled “Real-Time
>>>>>> Mode in Apache Spark Structured Streaming” that I've been working on with
>>>>>> Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [
>>>>>> JIRA <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>>> ].
>>>>>>
>>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
>>>>>> Spark Structured Streaming that significantly lowers end-to-end latency for
>>>>>> processing streams of data.
>>>>>>
>>>>>> A key principle of this proposal is compatibility. Our goal is to
>>>>>> make Spark capable of handling streaming jobs that need results almost
>>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>>> simply turning it on, without rewriting their logic.
>>>>>>
>>>>>> In short, we’re trying to enable Spark to power real-time
>>>>>> applications (like instant anomaly alerts or live personalization) that
>>>>>> today cannot meet their latency requirements with Spark’s current streaming
>>>>>> engine.
>>>>>>
>>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions on
>>>>>> this approach!
>>>>>>
>>>>>>
--
Best,
Yanbo
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by huaxin gao <hu...@gmail.com>.
+1
By unifying batch and low-latency streaming in Spark, we can eliminate the
need for separate streaming engines, reducing system complexity and
operational cost. Excited to see this direction!
On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi,
>
> My point about "in real time application or data, there is nothing as an
> answer which is supposed to be late and correct. The timeliness is part of
> the application. if I get the right answer too slowly it becomes useless or
> wrong" is actually fundamental to *why* we need this Spark Structured
> Streaming proposal.
>
> The proposal is precisely about enabling Spark to power applications
> where, as I define it, the *timeliness* of the answer is as critical as
> its *correctness*. Spark's current streaming engine, primarily operating
> on micro-batches, often delivers results that are technically "correct" but
> arrive too late to be truly useful for certain high-stakes, real-time
> scenarios. This makes them "useless or wrong" in a practical,
> business-critical sense.
>
> For example *in real-time fraud detection* and In *high-frequency
> trading,* market data or trade execution commands must be delivered with
> minimal latency. Even a slight delay can mean missed opportunities or
> significant financial losses, making a "correct" price update useless if
> it's not instantaneous. able for these demanding use cases, where a "late
> but correct" answer is simply not good enough. As a colliery it is a
> fundamental concept, so it has to be treated as such not as a comment.in
> SPIP
>
> Hope this clarifies the connection in practical terms
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com> wrote:
>
>> Hey Mich,
>>
>> Sorry, I may be missing something here but what does your definition here
>> have to do with the SPIP? Perhaps add comments directly to the SPIP to
>> provide context as the code snippet below is a direct copy from the SPIP
>> itself.
>>
>> Thanks,
>> Denny
>>
>>
>>
>>
>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> just to add
>>>
>>> A stronger definition of real time. The engineering definition of real
>>> time is roughly fast enough to be interactive
>>>
>>> However, I put a stronger definition. In real time application or data,
>>> there is nothing as an answer which is supposed to be late and correct. The
>>> timeliness is part of the application.if I get the right answer too slowly
>>> it becomes useless or wrong
>>>
>>>
>>>
>>> Dr Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>>> The current limitations in SSS come from micro-batching.If you are
>>>> going to reduce micro-batching, this reduction must be balanced against the
>>>> available processing capacity of the cluster to prevent back pressure and
>>>> instability. In the case of Continuous Processing mode, a specific
>>>> continuous trigger with a desired checkpoint interval quote
>>>>
>>>> "
>>>> df.writeStream
>>>> .format("...")
>>>> .option("...")
>>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type to
>>>> enable real-time Mode
>>>> .start()
>>>> This Trigger.RealTime signals that the query should run in the new
>>>> ultra low-latency execution mode. A time interval can also be specified,
>>>> e.g. “300 Seconds”, to indicate how long each micro-batch should run for.
>>>> "
>>>>
>>>> will inevitably depend on many factors. Not that simple
>>>> HTH
>>>>
>>>>
>>>> Dr Mich Talebzadeh,
>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <je...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I want to start a discussion thread for the SPIP titled “Real-Time
>>>>> Mode in Apache Spark Structured Streaming” that I've been working on with
>>>>> Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [
>>>>> JIRA <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>> ].
>>>>>
>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
>>>>> Spark Structured Streaming that significantly lowers end-to-end latency for
>>>>> processing streams of data.
>>>>>
>>>>> A key principle of this proposal is compatibility. Our goal is to make
>>>>> Spark capable of handling streaming jobs that need results almost
>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>> simply turning it on, without rewriting their logic.
>>>>>
>>>>> In short, we’re trying to enable Spark to power real-time applications
>>>>> (like instant anomaly alerts or live personalization) that today cannot
>>>>> meet their latency requirements with Spark’s current streaming engine.
>>>>>
>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions on
>>>>> this approach!
>>>>>
>>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
My point about "in real time application or data, there is nothing as an
answer which is supposed to be late and correct. The timeliness is part of
the application. if I get the right answer too slowly it becomes useless or
wrong" is actually fundamental to *why* we need this Spark Structured
Streaming proposal.
The proposal is precisely about enabling Spark to power applications where,
as I define it, the *timeliness* of the answer is as critical as its
*correctness*. Spark's current streaming engine, primarily operating on
micro-batches, often delivers results that are technically "correct" but
arrive too late to be truly useful for certain high-stakes, real-time
scenarios. This makes them "useless or wrong" in a practical,
business-critical sense.
For example *in real-time fraud detection* and In *high-frequency trading,*
market data or trade execution commands must be delivered with minimal
latency. Even a slight delay can mean missed opportunities or significant
financial losses, making a "correct" price update useless if it's not
instantaneous. able for these demanding use cases, where a "late but
correct" answer is simply not good enough. As a colliery it is a
fundamental concept, so it has to be treated as such not as a comment.in
SPIP
Hope this clarifies the connection in practical terms
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 28 May 2025 at 16:32, Denny Lee <de...@gmail.com> wrote:
> Hey Mich,
>
> Sorry, I may be missing something here but what does your definition here
> have to do with the SPIP? Perhaps add comments directly to the SPIP to
> provide context as the code snippet below is a direct copy from the SPIP
> itself.
>
> Thanks,
> Denny
>
>
>
>
> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> just to add
>>
>> A stronger definition of real time. The engineering definition of real
>> time is roughly fast enough to be interactive
>>
>> However, I put a stronger definition. In real time application or data,
>> there is nothing as an answer which is supposed to be late and correct. The
>> timeliness is part of the application.if I get the right answer too slowly
>> it becomes useless or wrong
>>
>>
>>
>> Dr Mich Talebzadeh,
>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> The current limitations in SSS come from micro-batching.If you are going
>>> to reduce micro-batching, this reduction must be balanced against the
>>> available processing capacity of the cluster to prevent back pressure and
>>> instability. In the case of Continuous Processing mode, a specific
>>> continuous trigger with a desired checkpoint interval quote
>>>
>>> "
>>> df.writeStream
>>> .format("...")
>>> .option("...")
>>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type to
>>> enable real-time Mode
>>> .start()
>>> This Trigger.RealTime signals that the query should run in the new ultra
>>> low-latency execution mode. A time interval can also be specified, e.g.
>>> “300 Seconds”, to indicate how long each micro-batch should run for.
>>> "
>>>
>>> will inevitably depend on many factors. Not that simple
>>> HTH
>>>
>>>
>>> Dr Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <je...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I want to start a discussion thread for the SPIP titled “Real-Time Mode
>>>> in Apache Spark Structured Streaming” that I've been working on with Siying
>>>> Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [JIRA
>>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>> ].
>>>>
>>>> The SPIP proposes a new execution mode called “Real-time Mode” in Spark
>>>> Structured Streaming that significantly lowers end-to-end latency for
>>>> processing streams of data.
>>>>
>>>> A key principle of this proposal is compatibility. Our goal is to make
>>>> Spark capable of handling streaming jobs that need results almost
>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>> simply turning it on, without rewriting their logic.
>>>>
>>>> In short, we’re trying to enable Spark to power real-time applications
>>>> (like instant anomaly alerts or live personalization) that today cannot
>>>> meet their latency requirements with Spark’s current streaming engine.
>>>>
>>>> We'd greatly appreciate your feedback, thoughts, and suggestions on
>>>> this approach!
>>>>
>>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Denny Lee <de...@gmail.com>.
Hey Mich,
Sorry, I may be missing something here but what does your definition here
have to do with the SPIP? Perhaps add comments directly to the SPIP to
provide context as the code snippet below is a direct copy from the SPIP
itself.
Thanks,
Denny
On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <mi...@gmail.com>
wrote:
> just to add
>
> A stronger definition of real time. The engineering definition of real
> time is roughly fast enough to be interactive
>
> However, I put a stronger definition. In real time application or data,
> there is nothing as an answer which is supposed to be late and correct. The
> timeliness is part of the application.if I get the right answer too slowly
> it becomes useless or wrong
>
>
>
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> The current limitations in SSS come from micro-batching.If you are going
>> to reduce micro-batching, this reduction must be balanced against the
>> available processing capacity of the cluster to prevent back pressure and
>> instability. In the case of Continuous Processing mode, a specific
>> continuous trigger with a desired checkpoint interval quote
>>
>> "
>> df.writeStream
>> .format("...")
>> .option("...")
>> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type to
>> enable real-time Mode
>> .start()
>> This Trigger.RealTime signals that the query should run in the new ultra
>> low-latency execution mode. A time interval can also be specified, e.g.
>> “300 Seconds”, to indicate how long each micro-batch should run for.
>> "
>>
>> will inevitably depend on many factors. Not that simple
>> HTH
>>
>>
>> Dr Mich Talebzadeh,
>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Wed, 28 May 2025 at 05:13, Jerry Peng <je...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I want to start a discussion thread for the SPIP titled “Real-Time Mode
>>> in Apache Spark Structured Streaming” that I've been working on with Siying
>>> Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [JIRA
>>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>> ].
>>>
>>> The SPIP proposes a new execution mode called “Real-time Mode” in Spark
>>> Structured Streaming that significantly lowers end-to-end latency for
>>> processing streams of data.
>>>
>>> A key principle of this proposal is compatibility. Our goal is to make
>>> Spark capable of handling streaming jobs that need results almost
>>> immediately (within O(100) milliseconds). We want to achieve this without
>>> changing the high-level DataFrame/Dataset API that users already use – so
>>> existing streaming queries can run in this new ultra-low-latency mode by
>>> simply turning it on, without rewriting their logic.
>>>
>>> In short, we’re trying to enable Spark to power real-time applications
>>> (like instant anomaly alerts or live personalization) that today cannot
>>> meet their latency requirements with Spark’s current streaming engine.
>>>
>>> We'd greatly appreciate your feedback, thoughts, and suggestions on this
>>> approach!
>>>
>>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mich Talebzadeh <mi...@gmail.com>.
just to add
A stronger definition of real time. The engineering definition of real time
is roughly fast enough to be interactive
However, I put a stronger definition. In real time application or data,
there is nothing as an answer which is supposed to be late and correct. The
timeliness is part of the application.if I get the right answer too slowly
it becomes useless or wrong
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <mi...@gmail.com>
wrote:
> The current limitations in SSS come from micro-batching.If you are going
> to reduce micro-batching, this reduction must be balanced against the
> available processing capacity of the cluster to prevent back pressure and
> instability. In the case of Continuous Processing mode, a specific
> continuous trigger with a desired checkpoint interval quote
>
> "
> df.writeStream
> .format("...")
> .option("...")
> .trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type to
> enable real-time Mode
> .start()
> This Trigger.RealTime signals that the query should run in the new ultra
> low-latency execution mode. A time interval can also be specified, e.g.
> “300 Seconds”, to indicate how long each micro-batch should run for.
> "
>
> will inevitably depend on many factors. Not that simple
> HTH
>
>
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Wed, 28 May 2025 at 05:13, Jerry Peng <je...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I want to start a discussion thread for the SPIP titled “Real-Time Mode
>> in Apache Spark Structured Streaming” that I've been working on with Siying
>> Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [JIRA
>> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>> ].
>>
>> The SPIP proposes a new execution mode called “Real-time Mode” in Spark
>> Structured Streaming that significantly lowers end-to-end latency for
>> processing streams of data.
>>
>> A key principle of this proposal is compatibility. Our goal is to make
>> Spark capable of handling streaming jobs that need results almost
>> immediately (within O(100) milliseconds). We want to achieve this without
>> changing the high-level DataFrame/Dataset API that users already use – so
>> existing streaming queries can run in this new ultra-low-latency mode by
>> simply turning it on, without rewriting their logic.
>>
>> In short, we’re trying to enable Spark to power real-time applications
>> (like instant anomaly alerts or live personalization) that today cannot
>> meet their latency requirements with Spark’s current streaming engine.
>>
>> We'd greatly appreciate your feedback, thoughts, and suggestions on this
>> approach!
>>
>>
Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming
Posted by Mich Talebzadeh <mi...@gmail.com>.
The current limitations in SSS come from micro-batching.If you are going to
reduce micro-batching, this reduction must be balanced against the
available processing capacity of the cluster to prevent back pressure and
instability. In the case of Continuous Processing mode, a specific
continuous trigger with a desired checkpoint interval quote
"
df.writeStream
.format("...")
.option("...")
.trigger(Trigger.RealTime(“300 Seconds”)) // new trigger type to
enable real-time Mode
.start()
This Trigger.RealTime signals that the query should run in the new ultra
low-latency execution mode. A time interval can also be specified, e.g.
“300 Seconds”, to indicate how long each micro-batch should run for.
"
will inevitably depend on many factors. Not that simple
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 28 May 2025 at 05:13, Jerry Peng <je...@gmail.com>
wrote:
> Hi all,
>
> I want to start a discussion thread for the SPIP titled “Real-Time Mode in
> Apache Spark Structured Streaming” that I've been working on with Siying
> Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [JIRA
> <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
> ].
>
> The SPIP proposes a new execution mode called “Real-time Mode” in Spark
> Structured Streaming that significantly lowers end-to-end latency for
> processing streams of data.
>
> A key principle of this proposal is compatibility. Our goal is to make
> Spark capable of handling streaming jobs that need results almost
> immediately (within O(100) milliseconds). We want to achieve this without
> changing the high-level DataFrame/Dataset API that users already use – so
> existing streaming queries can run in this new ultra-low-latency mode by
> simply turning it on, without rewriting their logic.
>
> In short, we’re trying to enable Spark to power real-time applications
> (like instant anomaly alerts or live personalization) that today cannot
> meet their latency requirements with Spark’s current streaming engine.
>
> We'd greatly appreciate your feedback, thoughts, and suggestions on this
> approach!
>
>