streaming state

[Alex Rovner <al...@magnetic.com>]

Hello all,

I was wondering if someone would be kind enough to enlighten me on a few
topics. We are trying to join two streams of data on a key. We were
thinking of partitioning topics in Kafka by the key, however I also saw
that Flink is able to partition on its own and I was wondering whether
Flink can take advantage of Kafka's partitioning and/or which partitioning
scheme should I go for?

As far as joins, our two datasets are very large (millions of records in
each window) and we need to perform the joins very quickly (less than 1
sec). Would the built in join mechanism be sufficient for this? From what I
understand it would need to shuffle data and thus be slow on large
datasets? I was wondering if there is a way to join via state key value
lookups to avoid the shuffling?

I read the docs and the blogs so far, thus have some limited understanding
of how Flink works, no practical experience though.

Thanks


*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052

* <http://www.magnetic.com/>*

Re: streaming state

[Alex Rovner <al...@magnetic.com>]

Thank you Stephan for the information!

On Mon, Dec 14, 2015 at 5:20 AM Stephan Ewen <se...@apache.org> wrote:

> Hi Alex!
>
> Right now, Flink would not reuse Kafka's partitioning for joins, but
> shuffle/partition data by itself. Flink is very fast at shuffling and adds
> very little latency on shuffles, so that is usually not an issue. The
> reason that design is that we view streaming program as something dynamic:
> Kafka partitions may be added or removed during a program's life time, and
> the parallelism (and with that the partitioning scheme) can change as well.
> With Flink handling the partitioning internally, these cases are all
> covered.
>
> Concerning the join: The built-in join is definitely able to handle
> millions or records in a window, and scales well. What it does is windowing
> the streams together and joining within the windows. If you want responses
> within a second, you should make the window small enough that it evaluated
> every 500ms or so.
>
> If you want super low latency joins, you can look into using custom state
> to do that. With that, you could build your custom symmetric hash join for
> example. That has virtually zero latency and you can control how long each
> side keeps the data.
>
> Concerning key lookups vs shuffling: The shuffle variant is usually much
> faster, because it uses network better. The shuffle is fully pipelined,
> many records are in shuffle at the same time, it is optimized for
> throughput and can still keep the latency quite low (
> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
> ).
> In contrast, key lookups that avoid shuffling usually take a bit of time
> (millisecond or so) and limit any throughput a lot because they involve
> many smaller messages and add even more latency (roundtrip between nodes,
> rather than one way).
>
>
> Hope that this answers your question, let me know if you have more
> questions!
>
> Greetings,
> Stephan
>
>
> On Fri, Dec 11, 2015 at 4:00 PM, Alex Rovner <al...@magnetic.com>
> wrote:
>
>> Hello all,
>>
>> I was wondering if someone would be kind enough to enlighten me on a few
>> topics. We are trying to join two streams of data on a key. We were
>> thinking of partitioning topics in Kafka by the key, however I also saw
>> that Flink is able to partition on its own and I was wondering whether
>> Flink can take advantage of Kafka's partitioning and/or which partitioning
>> scheme should I go for?
>>
>> As far as joins, our two datasets are very large (millions of records in
>> each window) and we need to perform the joins very quickly (less than 1
>> sec). Would the built in join mechanism be sufficient for this? From what I
>> understand it would need to shuffle data and thus be slow on large
>> datasets? I was wondering if there is a way to join via state key value
>> lookups to avoid the shuffling?
>>
>> I read the docs and the blogs so far, thus have some limited
>> understanding of how Flink works, no practical experience though.
>>
>> Thanks
>>
>>
>> *Alex Rovner*
>> *Director, Data Engineering *
>> *o:* 646.759.0052
>>
>> * <http://www.magnetic.com/>*
>>
>
> --
*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052

* <http://www.magnetic.com/>*

Re: streaming state

[Stephan Ewen <se...@apache.org>]

Hi Alex!

Right now, Flink would not reuse Kafka's partitioning for joins, but
shuffle/partition data by itself. Flink is very fast at shuffling and adds
very little latency on shuffles, so that is usually not an issue. The
reason that design is that we view streaming program as something dynamic:
Kafka partitions may be added or removed during a program's life time, and
the parallelism (and with that the partitioning scheme) can change as well.
With Flink handling the partitioning internally, these cases are all
covered.

Concerning the join: The built-in join is definitely able to handle
millions or records in a window, and scales well. What it does is windowing
the streams together and joining within the windows. If you want responses
within a second, you should make the window small enough that it evaluated
every 500ms or so.

If you want super low latency joins, you can look into using custom state
to do that. With that, you could build your custom symmetric hash join for
example. That has virtually zero latency and you can control how long each
side keeps the data.

Concerning key lookups vs shuffling: The shuffle variant is usually much
faster, because it uses network better. The shuffle is fully pipelined,
many records are in shuffle at the same time, it is optimized for
throughput and can still keep the latency quite low (
http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
).
In contrast, key lookups that avoid shuffling usually take a bit of time
(millisecond or so) and limit any throughput a lot because they involve
many smaller messages and add even more latency (roundtrip between nodes,
rather than one way).


Hope that this answers your question, let me know if you have more
questions!

Greetings,
Stephan


On Fri, Dec 11, 2015 at 4:00 PM, Alex Rovner <al...@magnetic.com>
wrote:

> Hello all,
>
> I was wondering if someone would be kind enough to enlighten me on a few
> topics. We are trying to join two streams of data on a key. We were
> thinking of partitioning topics in Kafka by the key, however I also saw
> that Flink is able to partition on its own and I was wondering whether
> Flink can take advantage of Kafka's partitioning and/or which partitioning
> scheme should I go for?
>
> As far as joins, our two datasets are very large (millions of records in
> each window) and we need to perform the joins very quickly (less than 1
> sec). Would the built in join mechanism be sufficient for this? From what I
> understand it would need to shuffle data and thus be slow on large
> datasets? I was wondering if there is a way to join via state key value
> lookups to avoid the shuffling?
>
> I read the docs and the blogs so far, thus have some limited understanding
> of how Flink works, no practical experience though.
>
> Thanks
>
>
> *Alex Rovner*
> *Director, Data Engineering *
> *o:* 646.759.0052
>
> * <http://www.magnetic.com/>*
>