You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Igal Shilman <ig...@ververica.com> on 2019/11/01 11:20:55 UTC

Re: Stateful functions presentation code (UI part)

Hi Flavio, let me try to clarify:

The intention of this example is to demonstrate how
different entities (drivers, passengers, etc') participates in a protocol
(ride matching). For that we have the stateful functions application, and a
standalone java application that just generates the events to trigger the
simulation (i.e. a passenger requests a ride would be an event that the
simulator emits).
The visualization aspect of this example is not a part of what we are
trying to demonstrate but rather, an out-of-band peek into the live system
(websocket that duplicates the events directly to the UI) which we thought
is a nice addition to the example that visualizes how cars are moving on
the grid :-)

If your goal is to create a scalable visualization of car locations, then I
guess your suggestion to break the grid into distinct geo-regions can work.
For that you would need:
1. An additional egress backed by a Kafka topic.
2. Modify the driver function to send location changes to that egress,
where you'd use the region id (geohash with a resolution to your liking) as
key [1].
3. At the consuming side subscribe to the partition that holds the region
of interest.

But, realistically speaking for this kind of a visualization you would be
better off consuming from the locations update topic (from-driver topic)
and populating an optimized geo-indexed store,
and querying it directly.

I hope this clarifies things,
Igal

[1]
https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121

On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier <po...@okkam.it>
wrote:

> Thanks Igal, this is more or less what I was expecting..this implies that
> ALL events are received on the UI side.
> I was concerned about the tradeoffs of this choice: when I zoom on the map
> I could simply ignore messages outside the boundaries (but I still spend
> many cpu resource in the reading of useless messages).
>
> In the case of a worldwide company (eg uber or simolar) it's probably better
> to create a topic per geographical arean..but also in this case then the UI
> should know when to attach or detach from queue topics when it reach a
> section of the map served by different topics..in another case I could just
> have too many events also in the single map section (let's think about some
> big city with many user).
>
> Is there any talk at FF ir someone else that faced those issues too?
>
> Il Gio 31 Ott 2019, 17:44 Igal Shilman <ig...@ververica.com> ha scritto:
>
>> For that particular example, the simulator [1] is responsible for
>> simulating physical drivers and passengers that interact with their
>> corresponding
>> stateful functions [2].
>> The interaction between the simulator and the stateful functions is
>> happening via four Kafka topics:
>> * to-driver - messages that are sent from the physical drivers (the
>> simulator [1]) to the stateful functions. The messages always carry a
>> driver-id which acts as a routing key (I think that this is what you mean
>> by correlation id) to a specific driver stateful function (FnDriver)
>> * from-driver - messages that are sent from a stateful function with a
>> specific driver id to the simulator
>> * to-passenger - symmetric to to-driver
>> * from-passenger - symmetric to from-driver.
>> The ingress and egress definition are specified here [3], and you may
>> want to checkout how to router is defined as well [4][5].
>>
>> In addition the simulator is also feeding the UI directly by duplicating
>> the messages to a web socket (see [6])
>>
>> I hope this clarifies the examples.
>>
>> Igal.
>>
>> [1]
>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator
>> [2]
>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions
>> [3]
>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43
>> [4]
>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33
>> [5]
>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26
>> [6]
>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70
>>
>> On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <po...@okkam.it>
>> wrote:
>>
>>> Yes, I'm interested in how to read data from a UI..which egress should I
>>> use? If we use a kafka queue, how to filter data received in the topic?
>>> Should I use a correlation id or use a new topic per user?
>>>
>>> Il Gio 31 Ott 2019, 16:08 Igal Shilman <ig...@ververica.com> ha scritto:
>>>
>>>> Hi Flavio,
>>>>
>>>> We haven't included the UI source code just yet, we've only used it for
>>>> demos and talks.
>>>>
>>>> The reason is that (1) we didn't put a lot of effort and time there (2)
>>>> didn't check the time to go through the individual dependencies and
>>>> licences.
>>>> But we will add that very soon.
>>>>
>>>> Would having the UI code there would improve your understanding? or is
>>>> there another reason?
>>>>
>>>> Thanks,
>>>> Igal
>>>>
>>>> On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <
>>>> pompermaier@okkam.it> wrote:
>>>>
>>>>> Hi Vino,
>>>>> I already checked that code but I can't find the UI part :(
>>>>>
>>>>> On Thu, Oct 31, 2019 at 12:32 PM vino yang <ya...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Flavio,
>>>>>>
>>>>>> Please see this link.[1]
>>>>>>
>>>>>> Best,
>>>>>> Vino
>>>>>>
>>>>>> [1]:
>>>>>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example
>>>>>>
>>>>>> Flavio Pompermaier <po...@okkam.it> 于2019年10月31日周四 下午4:53写道:
>>>>>>
>>>>>>> Hi to all,
>>>>>>> yould it be possible to provide also the source code of the UI part
>>>>>>> of the ride sharing example? It would be interesting to me how the UI is
>>>>>>> reading the data from the Kafka egress.
>>>>>>>
>>>>>>> Best,
>>>>>>> Flavio
>>>>>>>
>>>>>>
>>>>>

Re: Stateful functions presentation code (UI part)

Posted by Flavio Pompermaier <po...@okkam.it>.
Thanks Igal for the detailed explanantion. I know that this was only a
demo, I just wanted to reason a bit on the pros and cons of sending data to
an UI from Flink.

Best,
Flavio


Il Ven 1 Nov 2019, 12:21 Igal Shilman <ig...@ververica.com> ha scritto:

> Hi Flavio, let me try to clarify:
>
> The intention of this example is to demonstrate how
> different entities (drivers, passengers, etc') participates in a protocol
> (ride matching). For that we have the stateful functions application, and a
> standalone java application that just generates the events to trigger the
> simulation (i.e. a passenger requests a ride would be an event that the
> simulator emits).
> The visualization aspect of this example is not a part of what we are
> trying to demonstrate but rather, an out-of-band peek into the live system
> (websocket that duplicates the events directly to the UI) which we thought
> is a nice addition to the example that visualizes how cars are moving on
> the grid :-)
>
> If your goal is to create a scalable visualization of car locations, then
> I guess your suggestion to break the grid into distinct geo-regions can
> work. For that you would need:
> 1. An additional egress backed by a Kafka topic.
> 2. Modify the driver function to send location changes to that egress,
> where you'd use the region id (geohash with a resolution to your liking) as
> key [1].
> 3. At the consuming side subscribe to the partition that holds the region
> of interest.
>
> But, realistically speaking for this kind of a visualization you would be
> better off consuming from the locations update topic (from-driver topic)
> and populating an optimized geo-indexed store,
> and querying it directly.
>
> I hope this clarifies things,
> Igal
>
> [1]
> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121
>
> On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Thanks Igal, this is more or less what I was expecting..this implies that
>> ALL events are received on the UI side.
>> I was concerned about the tradeoffs of this choice: when I zoom on the
>> map I could simply ignore messages outside the boundaries (but I still
>> spend many cpu resource in the reading of useless messages).
>>
>> In the case of a worldwide company (eg uber or simolar) it's probably better
>> to create a topic per geographical arean..but also in this case then the UI
>> should know when to attach or detach from queue topics when it reach a
>> section of the map served by different topics..in another case I could just
>> have too many events also in the single map section (let's think about some
>> big city with many user).
>>
>> Is there any talk at FF ir someone else that faced those issues too?
>>
>> Il Gio 31 Ott 2019, 17:44 Igal Shilman <ig...@ververica.com> ha scritto:
>>
>>> For that particular example, the simulator [1] is responsible for
>>> simulating physical drivers and passengers that interact with their
>>> corresponding
>>> stateful functions [2].
>>> The interaction between the simulator and the stateful functions is
>>> happening via four Kafka topics:
>>> * to-driver - messages that are sent from the physical drivers (the
>>> simulator [1]) to the stateful functions. The messages always carry a
>>> driver-id which acts as a routing key (I think that this is what you mean
>>> by correlation id) to a specific driver stateful function (FnDriver)
>>> * from-driver - messages that are sent from a stateful function with a
>>> specific driver id to the simulator
>>> * to-passenger - symmetric to to-driver
>>> * from-passenger - symmetric to from-driver.
>>> The ingress and egress definition are specified here [3], and you may
>>> want to checkout how to router is defined as well [4][5].
>>>
>>> In addition the simulator is also feeding the UI directly by duplicating
>>> the messages to a web socket (see [6])
>>>
>>> I hope this clarifies the examples.
>>>
>>> Igal.
>>>
>>> [1]
>>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator
>>> [2]
>>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions
>>> [3]
>>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43
>>> [4]
>>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33
>>> [5]
>>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26
>>> [6]
>>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70
>>>
>>> On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <po...@okkam.it>
>>> wrote:
>>>
>>>> Yes, I'm interested in how to read data from a UI..which egress should
>>>> I use? If we use a kafka queue, how to filter data received in the topic?
>>>> Should I use a correlation id or use a new topic per user?
>>>>
>>>> Il Gio 31 Ott 2019, 16:08 Igal Shilman <ig...@ververica.com> ha scritto:
>>>>
>>>>> Hi Flavio,
>>>>>
>>>>> We haven't included the UI source code just yet, we've only used it
>>>>> for demos and talks.
>>>>>
>>>>> The reason is that (1) we didn't put a lot of effort and time there
>>>>> (2) didn't check the time to go through the individual dependencies and
>>>>> licences.
>>>>> But we will add that very soon.
>>>>>
>>>>> Would having the UI code there would improve your understanding? or is
>>>>> there another reason?
>>>>>
>>>>> Thanks,
>>>>> Igal
>>>>>
>>>>> On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <
>>>>> pompermaier@okkam.it> wrote:
>>>>>
>>>>>> Hi Vino,
>>>>>> I already checked that code but I can't find the UI part :(
>>>>>>
>>>>>> On Thu, Oct 31, 2019 at 12:32 PM vino yang <ya...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Flavio,
>>>>>>>
>>>>>>> Please see this link.[1]
>>>>>>>
>>>>>>> Best,
>>>>>>> Vino
>>>>>>>
>>>>>>> [1]:
>>>>>>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example
>>>>>>>
>>>>>>> Flavio Pompermaier <po...@okkam.it> 于2019年10月31日周四 下午4:53写道:
>>>>>>>
>>>>>>>> Hi to all,
>>>>>>>> yould it be possible to provide also the source code of the UI part
>>>>>>>> of the ride sharing example? It would be interesting to me how the UI is
>>>>>>>> reading the data from the Kafka egress.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Flavio
>>>>>>>>
>>>>>>>
>>>>>>