You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stephan Ewen <se...@apache.org> on 2015/08/05 16:11:29 UTC
FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Hi all!
We just published a blog post about how streaming fault tolerance
mechanisms evolved, and what kind of performance Flink gets with its
checkpointing mechanism.
I think it is a pretty interesting read for people that are interested in
Flink or data streaming in general.
The blog post talks about:
- Fault tolerance techniques, starting from acknowledgements, over micro
batches, to transactional updates and distributed snapshots.
- Performance of Flink, throughput, latency, and tradeoffs.
- A "chaos monkey" experiment where computation continues strongly
consistent even when periodically killing workers.
Comments welcome!
Greetings,
Stephan
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Stephan Ewen <se...@apache.org>.
Thanks, Theo.
Fixed the markup...
On Wed, Aug 5, 2015 at 11:08 PM, Theodore Vasiloudis <
theodoros.vasiloudis@gmail.com> wrote:
> Great post Stephan! A small note: the code for Google Dataflow does display
> correctly for me, I'm getting < and > instead of <>
>
> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > Hi all!
> >
> > We just published a blog post about how streaming fault tolerance
> > mechanisms evolved, and what kind of performance Flink gets with its
> > checkpointing mechanism.
> >
> > I think it is a pretty interesting read for people that are interested in
> > Flink or data streaming in general.
> >
> > The blog post talks about:
> >
> > - Fault tolerance techniques, starting from acknowledgements, over
> micro
> > batches, to transactional updates and distributed snapshots.
> >
> > - Performance of Flink, throughput, latency, and tradeoffs.
> >
> > - A "chaos monkey" experiment where computation continues strongly
> > consistent even when periodically killing workers.
> >
> >
> > Comments welcome!
> >
> > Greetings,
> > Stephan
> >
>
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Sachin Goel <sa...@gmail.com>.
This makes me want to dig into the streaming API. About time perhaps. :')
Cheers!
Sachin
-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685
On Aug 6, 2015 2:39 AM, "Theodore Vasiloudis" <
theodoros.vasiloudis@gmail.com> wrote:
> Great post Stephan! A small note: the code for Google Dataflow does display
> correctly for me, I'm getting < and > instead of <>
>
> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > Hi all!
> >
> > We just published a blog post about how streaming fault tolerance
> > mechanisms evolved, and what kind of performance Flink gets with its
> > checkpointing mechanism.
> >
> > I think it is a pretty interesting read for people that are interested in
> > Flink or data streaming in general.
> >
> > The blog post talks about:
> >
> > - Fault tolerance techniques, starting from acknowledgements, over
> micro
> > batches, to transactional updates and distributed snapshots.
> >
> > - Performance of Flink, throughput, latency, and tradeoffs.
> >
> > - A "chaos monkey" experiment where computation continues strongly
> > consistent even when periodically killing workers.
> >
> >
> > Comments welcome!
> >
> > Greetings,
> > Stephan
> >
>
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Theodore Vasiloudis <th...@gmail.com>.
Great post Stephan! A small note: the code for Google Dataflow does display
correctly for me, I'm getting < and > instead of <>
On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi all!
>
> We just published a blog post about how streaming fault tolerance
> mechanisms evolved, and what kind of performance Flink gets with its
> checkpointing mechanism.
>
> I think it is a pretty interesting read for people that are interested in
> Flink or data streaming in general.
>
> The blog post talks about:
>
> - Fault tolerance techniques, starting from acknowledgements, over micro
> batches, to transactional updates and distributed snapshots.
>
> - Performance of Flink, throughput, latency, and tradeoffs.
>
> - A "chaos monkey" experiment where computation continues strongly
> consistent even when periodically killing workers.
>
>
> Comments welcome!
>
> Greetings,
> Stephan
>
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Ankur Chauhan <an...@malloc64.com>.
Pretty awesome piece.
Sent from my iPhone
> On Aug 5, 2015, at 10:10, Hawin Jiang <ha...@gmail.com> wrote:
>
> Great job, Guys
>
> Let me read it carefully.
>
>
>
>
>
>
>
>> On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <se...@apache.org> wrote:
>> I forgot the link ;-)
>>
>> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
>>
>>> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
>>> Hi all!
>>>
>>> We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism.
>>>
>>> I think it is a pretty interesting read for people that are interested in Flink or data streaming in general.
>>>
>>> The blog post talks about:
>>>
>>> - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots.
>>>
>>> - Performance of Flink, throughput, latency, and tradeoffs.
>>>
>>> - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers.
>>>
>>>
>>> Comments welcome!
>>>
>>> Greetings,
>>> Stephan
>
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Ankur Chauhan <an...@malloc64.com>.
Pretty awesome piece.
Sent from my iPhone
> On Aug 5, 2015, at 10:10, Hawin Jiang <ha...@gmail.com> wrote:
>
> Great job, Guys
>
> Let me read it carefully.
>
>
>
>
>
>
>
>> On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <se...@apache.org> wrote:
>> I forgot the link ;-)
>>
>> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
>>
>>> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
>>> Hi all!
>>>
>>> We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism.
>>>
>>> I think it is a pretty interesting read for people that are interested in Flink or data streaming in general.
>>>
>>> The blog post talks about:
>>>
>>> - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots.
>>>
>>> - Performance of Flink, throughput, latency, and tradeoffs.
>>>
>>> - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers.
>>>
>>>
>>> Comments welcome!
>>>
>>> Greetings,
>>> Stephan
>
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Hawin Jiang <ha...@gmail.com>.
Great job, Guys
Let me read it carefully.
On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <se...@apache.org> wrote:
> I forgot the link ;-)
>
>
> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
>
> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi all!
>>
>> We just published a blog post about how streaming fault tolerance
>> mechanisms evolved, and what kind of performance Flink gets with its
>> checkpointing mechanism.
>>
>> I think it is a pretty interesting read for people that are interested in
>> Flink or data streaming in general.
>>
>> The blog post talks about:
>>
>> - Fault tolerance techniques, starting from acknowledgements, over
>> micro batches, to transactional updates and distributed snapshots.
>>
>> - Performance of Flink, throughput, latency, and tradeoffs.
>>
>> - A "chaos monkey" experiment where computation continues strongly
>> consistent even when periodically killing workers.
>>
>>
>> Comments welcome!
>>
>> Greetings,
>> Stephan
>>
>>
>>
>
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Hawin Jiang <ha...@gmail.com>.
Great job, Guys
Let me read it carefully.
On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <se...@apache.org> wrote:
> I forgot the link ;-)
>
>
> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
>
> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi all!
>>
>> We just published a blog post about how streaming fault tolerance
>> mechanisms evolved, and what kind of performance Flink gets with its
>> checkpointing mechanism.
>>
>> I think it is a pretty interesting read for people that are interested in
>> Flink or data streaming in general.
>>
>> The blog post talks about:
>>
>> - Fault tolerance techniques, starting from acknowledgements, over
>> micro batches, to transactional updates and distributed snapshots.
>>
>> - Performance of Flink, throughput, latency, and tradeoffs.
>>
>> - A "chaos monkey" experiment where computation continues strongly
>> consistent even when periodically killing workers.
>>
>>
>> Comments welcome!
>>
>> Greetings,
>> Stephan
>>
>>
>>
>
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Stephan Ewen <se...@apache.org>.
I forgot the link ;-)
http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi all!
>
> We just published a blog post about how streaming fault tolerance
> mechanisms evolved, and what kind of performance Flink gets with its
> checkpointing mechanism.
>
> I think it is a pretty interesting read for people that are interested in
> Flink or data streaming in general.
>
> The blog post talks about:
>
> - Fault tolerance techniques, starting from acknowledgements, over micro
> batches, to transactional updates and distributed snapshots.
>
> - Performance of Flink, throughput, latency, and tradeoffs.
>
> - A "chaos monkey" experiment where computation continues strongly
> consistent even when periodically killing workers.
>
>
> Comments welcome!
>
> Greetings,
> Stephan
>
>
>
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Posted by Stephan Ewen <se...@apache.org>.
I forgot the link ;-)
http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi all!
>
> We just published a blog post about how streaming fault tolerance
> mechanisms evolved, and what kind of performance Flink gets with its
> checkpointing mechanism.
>
> I think it is a pretty interesting read for people that are interested in
> Flink or data streaming in general.
>
> The blog post talks about:
>
> - Fault tolerance techniques, starting from acknowledgements, over micro
> batches, to transactional updates and distributed snapshots.
>
> - Performance of Flink, throughput, latency, and tradeoffs.
>
> - A "chaos monkey" experiment where computation continues strongly
> consistent even when periodically killing workers.
>
>
> Comments welcome!
>
> Greetings,
> Stephan
>
>
>