You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Felipe Gutierrez <fe...@gmail.com> on 2019/10/10 12:39:59 UTC
Difference between windows in Spark and Flink
Hi all,
I am trying to think about the essential differences between operators in
Flink and Spark. Especially when I am using Keyed Windows then a reduce
operation.
In Flink we develop an application that can logically separate these two
operators. Thus after a keyed window I can use
.reduce/aggregate/fold/apply() functions [1].
In Spark we have window/reduceByKeyAndWindow functions which to me appears
it is less flexible in the options to use with a keyed window operation [2].
Moreover, when these two applications are deployed in a Flink and Spark
cluster respectively, what are the differences between their physical
operators running in the cluster?
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#windows
[2]
https://spark.apache.org/docs/latest/streaming-programming-guide.html#window-operations
Thanks,
Felipe
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
Re: Difference between windows in Spark and Flink
Posted by Felipe Gutierrez <fe...@gmail.com>.
that is nice. So, only by this Flink shuffles fewer data them Spark.
Now I need to plug Prometheus and Grafana to show it.
Thanks Yun for your help!
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
On Fri, Oct 11, 2019 at 12:08 PM Yun Tang <my...@live.com> wrote:
> Hi Felipe
>
> From what I remember, Spark still use micro-batch to shuffle data in
> structed streaming.
> For Flink, it actually process elements per record, there is no actual
> disk-io shuffle in Flink streaming. And record would emit to downstream by
> select specific channel through network[1]. That's why we need to call "
> keyBy" before using windows, "KeyGroupStreamPartitioner" would then be
> used to select the target channel based on the key group index. Data would
> first be stored in local state backend and wait for polled out once a
> window triggered but not "shuffled" until a window triggered.
>
>
> [1]
> https://github.com/apache/flink/blob/2c2095bdad3d47f27973a585112ed820f457de6f/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ChannelSelectorRecordWriter.java#L60
>
> Best
> Yun Tang
>
> ------------------------------
> *From:* Felipe Gutierrez <fe...@gmail.com>
> *Sent:* Friday, October 11, 2019 15:47
> *To:* Yun Tang <my...@live.com>
> *Cc:* user <us...@flink.apache.org>
> *Subject:* Re: Difference between windows in Spark and Flink
>
> Hi Yun,
>
> that is a very complete answer. Thanks!
>
> I was also wondering about the mini-batches that Spark creates when we
> have to create a SparkStream context. It still remains for all versions of
> stream processing in Spark, isn't it? And because that I Spark shuffles
> data [1] to wide-dependent operators every time mini-batch ends, doesn't it?
> In this way Flink does not have mini-batches, hence I will shuffle data to
> wide-dependent operators only when a window is triggered. Am I correct?
>
> [1]
> http://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/rdd-programming-guide.html#shuffle-operations
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez *
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
>
> On Thu, Oct 10, 2019 at 7:25 PM Yun Tang <my...@live.com> wrote:
>
> Hi Felipe
>
> Generally speaking, the key difference which impacts the performance is
> where they store data within windows.
> For Flink, it would store data and its related time-stamp within windows
> in state backend[1]. Once window is triggered, it would pull all the stored
> timer with coupled record-key, and then use the record-key to query state
> backend for next actions.
>
> For Spark, first of all, we would talk about structured streaming [2] as
> it's better than previous spark streaming especially on window scenario.
> Unlike Flink built-in supported rocksDB state backend, Spark has only one
> implementation of state store providers. It's HDFSBackedStateStoreProvider
> which stores all of the data in memory, what is a very memory consuming
> approach and might come across OOM errors[3][4][5].
>
> To avoid this, Databricks Runtime offer a 'RocksDBStateStoreProvider' but
> not open-source. We're lucky that open-source Flink already offers built-in
> RocksDB state backend to avoid OOM problem. Moreover, Flink community
> recently are developing spill-able memory state backend [7].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html
> [2]
> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time
> [3]
> https://medium.com/@chandanbaranwal/state-management-in-spark-structured-streaming-aaa87b6c9d31
> [4]
> http://apache-spark-user-list.1001560.n3.nabble.com/use-rocksdb-for-spark-structured-streaming-SSS-td34776.html#a34779
> [5] https://github.com/chermenin/spark-states
> [6]
> https://docs.databricks.com/spark/latest/structured-streaming/production.html#optimize-performance-of-stateful-streaming-queries
> [7] https://issues.apache.org/jira/browse/FLINK-12692
>
> Best
> Yun Tang
>
>
>
> ------------------------------
> *From:* Felipe Gutierrez <fe...@gmail.com>
> *Sent:* Thursday, October 10, 2019 20:39
> *To:* user <us...@flink.apache.org>
> *Subject:* Difference between windows in Spark and Flink
>
> Hi all,
>
> I am trying to think about the essential differences between operators in
> Flink and Spark. Especially when I am using Keyed Windows then a reduce
> operation.
> In Flink we develop an application that can logically separate these two
> operators. Thus after a keyed window I can use
> .reduce/aggregate/fold/apply() functions [1].
> In Spark we have window/reduceByKeyAndWindow functions which to me appears
> it is less flexible in the options to use with a keyed window operation [2].
> Moreover, when these two applications are deployed in a Flink and Spark
> cluster respectively, what are the differences between their physical
> operators running in the cluster?
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#windows
> [2]
> https://spark.apache.org/docs/latest/streaming-programming-guide.html#window-operations
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez *
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
>
Re: Difference between windows in Spark and Flink
Posted by Yun Tang <my...@live.com>.
Hi Felipe
From what I remember, Spark still use micro-batch to shuffle data in structed streaming.
For Flink, it actually process elements per record, there is no actual disk-io shuffle in Flink streaming. And record would emit to downstream by select specific channel through network[1]. That's why we need to call "keyBy" before using windows, "KeyGroupStreamPartitioner" would then be used to select the target channel based on the key group index. Data would first be stored in local state backend and wait for polled out once a window triggered but not "shuffled" until a window triggered.
[1] https://github.com/apache/flink/blob/2c2095bdad3d47f27973a585112ed820f457de6f/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ChannelSelectorRecordWriter.java#L60
Best
Yun Tang
________________________________
From: Felipe Gutierrez <fe...@gmail.com>
Sent: Friday, October 11, 2019 15:47
To: Yun Tang <my...@live.com>
Cc: user <us...@flink.apache.org>
Subject: Re: Difference between windows in Spark and Flink
Hi Yun,
that is a very complete answer. Thanks!
I was also wondering about the mini-batches that Spark creates when we have to create a SparkStream context. It still remains for all versions of stream processing in Spark, isn't it? And because that I Spark shuffles data [1] to wide-dependent operators every time mini-batch ends, doesn't it?
In this way Flink does not have mini-batches, hence I will shuffle data to wide-dependent operators only when a window is triggered. Am I correct?
[1] http://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/rdd-programming-guide.html#shuffle-operations
Thanks,
Felipe
--
-- Felipe Gutierrez
-- skype: felipe.o.gutierrez
-- https://felipeogutierrez.blogspot.com
On Thu, Oct 10, 2019 at 7:25 PM Yun Tang <my...@live.com>> wrote:
Hi Felipe
Generally speaking, the key difference which impacts the performance is where they store data within windows.
For Flink, it would store data and its related time-stamp within windows in state backend[1]. Once window is triggered, it would pull all the stored timer with coupled record-key, and then use the record-key to query state backend for next actions.
For Spark, first of all, we would talk about structured streaming [2] as it's better than previous spark streaming especially on window scenario. Unlike Flink built-in supported rocksDB state backend, Spark has only one implementation of state store providers. It's HDFSBackedStateStoreProvider which stores all of the data in memory, what is a very memory consuming approach and might come across OOM errors[3][4][5].
To avoid this, Databricks Runtime offer a 'RocksDBStateStoreProvider' but not open-source. We're lucky that open-source Flink already offers built-in RocksDB state backend to avoid OOM problem. Moreover, Flink community recently are developing spill-able memory state backend [7].
[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html
[2] https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time
[3] https://medium.com/@chandanbaranwal/state-management-in-spark-structured-streaming-aaa87b6c9d31
[4] http://apache-spark-user-list.1001560.n3.nabble.com/use-rocksdb-for-spark-structured-streaming-SSS-td34776.html#a34779
[5] https://github.com/chermenin/spark-states
[6] https://docs.databricks.com/spark/latest/structured-streaming/production.html#optimize-performance-of-stateful-streaming-queries
[7] https://issues.apache.org/jira/browse/FLINK-12692
Best
Yun Tang
________________________________
From: Felipe Gutierrez <fe...@gmail.com>>
Sent: Thursday, October 10, 2019 20:39
To: user <us...@flink.apache.org>>
Subject: Difference between windows in Spark and Flink
Hi all,
I am trying to think about the essential differences between operators in Flink and Spark. Especially when I am using Keyed Windows then a reduce operation.
In Flink we develop an application that can logically separate these two operators. Thus after a keyed window I can use .reduce/aggregate/fold/apply() functions [1].
In Spark we have window/reduceByKeyAndWindow functions which to me appears it is less flexible in the options to use with a keyed window operation [2].
Moreover, when these two applications are deployed in a Flink and Spark cluster respectively, what are the differences between their physical operators running in the cluster?
[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#windows
[2] https://spark.apache.org/docs/latest/streaming-programming-guide.html#window-operations
Thanks,
Felipe
--
-- Felipe Gutierrez
-- skype: felipe.o.gutierrez
-- https://felipeogutierrez.blogspot.com
Re: Difference between windows in Spark and Flink
Posted by Felipe Gutierrez <fe...@gmail.com>.
Hi Yun,
that is a very complete answer. Thanks!
I was also wondering about the mini-batches that Spark creates when we have
to create a SparkStream context. It still remains for all versions of
stream processing in Spark, isn't it? And because that I Spark shuffles
data [1] to wide-dependent operators every time mini-batch ends, doesn't it?
In this way Flink does not have mini-batches, hence I will shuffle data to
wide-dependent operators only when a window is triggered. Am I correct?
[1]
http://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/rdd-programming-guide.html#shuffle-operations
Thanks,
Felipe
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
On Thu, Oct 10, 2019 at 7:25 PM Yun Tang <my...@live.com> wrote:
> Hi Felipe
>
> Generally speaking, the key difference which impacts the performance is
> where they store data within windows.
> For Flink, it would store data and its related time-stamp within windows
> in state backend[1]. Once window is triggered, it would pull all the stored
> timer with coupled record-key, and then use the record-key to query state
> backend for next actions.
>
> For Spark, first of all, we would talk about structured streaming [2] as
> it's better than previous spark streaming especially on window scenario.
> Unlike Flink built-in supported rocksDB state backend, Spark has only one
> implementation of state store providers. It's HDFSBackedStateStoreProvider
> which stores all of the data in memory, what is a very memory consuming
> approach and might come across OOM errors[3][4][5].
>
> To avoid this, Databricks Runtime offer a 'RocksDBStateStoreProvider' but
> not open-source. We're lucky that open-source Flink already offers built-in
> RocksDB state backend to avoid OOM problem. Moreover, Flink community
> recently are developing spill-able memory state backend [7].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html
> [2]
> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time
> [3]
> https://medium.com/@chandanbaranwal/state-management-in-spark-structured-streaming-aaa87b6c9d31
> [4]
> http://apache-spark-user-list.1001560.n3.nabble.com/use-rocksdb-for-spark-structured-streaming-SSS-td34776.html#a34779
> [5] https://github.com/chermenin/spark-states
> [6]
> https://docs.databricks.com/spark/latest/structured-streaming/production.html#optimize-performance-of-stateful-streaming-queries
> [7] https://issues.apache.org/jira/browse/FLINK-12692
>
> Best
> Yun Tang
>
>
>
> ------------------------------
> *From:* Felipe Gutierrez <fe...@gmail.com>
> *Sent:* Thursday, October 10, 2019 20:39
> *To:* user <us...@flink.apache.org>
> *Subject:* Difference between windows in Spark and Flink
>
> Hi all,
>
> I am trying to think about the essential differences between operators in
> Flink and Spark. Especially when I am using Keyed Windows then a reduce
> operation.
> In Flink we develop an application that can logically separate these two
> operators. Thus after a keyed window I can use
> .reduce/aggregate/fold/apply() functions [1].
> In Spark we have window/reduceByKeyAndWindow functions which to me appears
> it is less flexible in the options to use with a keyed window operation [2].
> Moreover, when these two applications are deployed in a Flink and Spark
> cluster respectively, what are the differences between their physical
> operators running in the cluster?
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#windows
> [2]
> https://spark.apache.org/docs/latest/streaming-programming-guide.html#window-operations
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez *
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
Re: Difference between windows in Spark and Flink
Posted by Yun Tang <my...@live.com>.
Hi Felipe
Generally speaking, the key difference which impacts the performance is where they store data within windows.
For Flink, it would store data and its related time-stamp within windows in state backend[1]. Once window is triggered, it would pull all the stored timer with coupled record-key, and then use the record-key to query state backend for next actions.
For Spark, first of all, we would talk about structured streaming [2] as it's better than previous spark streaming especially on window scenario. Unlike Flink built-in supported rocksDB state backend, Spark has only one implementation of state store providers. It's HDFSBackedStateStoreProvider which stores all of the data in memory, what is a very memory consuming approach and might come across OOM errors[3][4][5].
To avoid this, Databricks Runtime offer a 'RocksDBStateStoreProvider' but not open-source. We're lucky that open-source Flink already offers built-in RocksDB state backend to avoid OOM problem. Moreover, Flink community recently are developing spill-able memory state backend [7].
[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html
[2] https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time
[3] https://medium.com/@chandanbaranwal/state-management-in-spark-structured-streaming-aaa87b6c9d31
[4] http://apache-spark-user-list.1001560.n3.nabble.com/use-rocksdb-for-spark-structured-streaming-SSS-td34776.html#a34779
[5] https://github.com/chermenin/spark-states
[6] https://docs.databricks.com/spark/latest/structured-streaming/production.html#optimize-performance-of-stateful-streaming-queries
[7] https://issues.apache.org/jira/browse/FLINK-12692
Best
Yun Tang
________________________________
From: Felipe Gutierrez <fe...@gmail.com>
Sent: Thursday, October 10, 2019 20:39
To: user <us...@flink.apache.org>
Subject: Difference between windows in Spark and Flink
Hi all,
I am trying to think about the essential differences between operators in Flink and Spark. Especially when I am using Keyed Windows then a reduce operation.
In Flink we develop an application that can logically separate these two operators. Thus after a keyed window I can use .reduce/aggregate/fold/apply() functions [1].
In Spark we have window/reduceByKeyAndWindow functions which to me appears it is less flexible in the options to use with a keyed window operation [2].
Moreover, when these two applications are deployed in a Flink and Spark cluster respectively, what are the differences between their physical operators running in the cluster?
[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#windows
[2] https://spark.apache.org/docs/latest/streaming-programming-guide.html#window-operations
Thanks,
Felipe
--
-- Felipe Gutierrez
-- skype: felipe.o.gutierrez
-- https://felipeogutierrez.blogspot.com