Dataflow worker overview graphs

[Mikhail Gryzykhin <mi...@google.com>]

Hello everybody,

Just wanted to share that I have found some graphs for dataflow worker I
created while starting working on it. They cover specific scenarios, but
may be useful for newcomers, so I put them into this wiki page
<https://cwiki.apache.org/confluence/display/BEAM/Dataflow+Worker+overview+graphs>
.

If you feel they belong to some other location, please let me know.

Regards,
Mikhail.

Re: Dataflow worker overview graphs

[Reza Rokni <re...@google.com>]

Very nice resource, thanx Mikhail.



On Fri, 9 Aug 2019 at 05:22, Mikhail Gryzykhin <mi...@google.com> wrote:

> Unfortunately no, I don't have those for streaming explicitly.
>
> However most of code is shared between streaming and batch with main
> difference in initialization. Same goes for boilerplate parts of legacy vs
> FnApi.
>
> If you happen to create anything similar for streaming, please update page
> and let me know. Also I'll update this page with relevant changes once I
> get back to worker.
>
> --Mikhail
>
> On Thu, Aug 8, 2019 at 2:13 PM Ankur Goenka <go...@google.com> wrote:
>
>> Thanks Mikhail. This is really useful.
>> Do you also have something similar for Streaming use case. More
>> specifically for Portable (fn_api) based streaming pipelines.
>>
>>
>> On Thu, Aug 8, 2019 at 2:08 PM Mikhail Gryzykhin <mi...@google.com>
>> wrote:
>>
>>> Hello everybody,
>>>
>>> Just wanted to share that I have found some graphs for dataflow worker I
>>> created while starting working on it. They cover specific scenarios, but
>>> may be useful for newcomers, so I put them into this wiki page
>>> <https://cwiki.apache.org/confluence/display/BEAM/Dataflow+Worker+overview+graphs>
>>> .
>>>
>>> If you feel they belong to some other location, please let me know.
>>>
>>> Regards,
>>> Mikhail.
>>>
>>

-- 

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

Re: Dataflow worker overview graphs

[Mikhail Gryzykhin <mi...@google.com>]

Unfortunately no, I don't have those for streaming explicitly.

However most of code is shared between streaming and batch with main
difference in initialization. Same goes for boilerplate parts of legacy vs
FnApi.

If you happen to create anything similar for streaming, please update page
and let me know. Also I'll update this page with relevant changes once I
get back to worker.

--Mikhail

On Thu, Aug 8, 2019 at 2:13 PM Ankur Goenka <go...@google.com> wrote:

> Thanks Mikhail. This is really useful.
> Do you also have something similar for Streaming use case. More
> specifically for Portable (fn_api) based streaming pipelines.
>
>
> On Thu, Aug 8, 2019 at 2:08 PM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Hello everybody,
>>
>> Just wanted to share that I have found some graphs for dataflow worker I
>> created while starting working on it. They cover specific scenarios, but
>> may be useful for newcomers, so I put them into this wiki page
>> <https://cwiki.apache.org/confluence/display/BEAM/Dataflow+Worker+overview+graphs>
>> .
>>
>> If you feel they belong to some other location, please let me know.
>>
>> Regards,
>> Mikhail.
>>
>

Re: Dataflow worker overview graphs

[Ankur Goenka <go...@google.com>]

Thanks Mikhail. This is really useful.
Do you also have something similar for Streaming use case. More
specifically for Portable (fn_api) based streaming pipelines.


On Thu, Aug 8, 2019 at 2:08 PM Mikhail Gryzykhin <mi...@google.com> wrote:

> Hello everybody,
>
> Just wanted to share that I have found some graphs for dataflow worker I
> created while starting working on it. They cover specific scenarios, but
> may be useful for newcomers, so I put them into this wiki page
> <https://cwiki.apache.org/confluence/display/BEAM/Dataflow+Worker+overview+graphs>
> .
>
> If you feel they belong to some other location, please let me know.
>
> Regards,
> Mikhail.
>