You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Pablo Estrada <pa...@google.com> on 2019/10/15 21:10:24 UTC
[design] A streaming Fn API runner for Python
Hello all,
I am planning to work on removing the old BundleBasedDirectRunner, and
expand the FnApiRunner to work on streaming as well as batch.
Currently, the FnApiRunner orders the processing graph topologically, and
"pushes" all the data through each stage in topological order (deferred
inputs such as residuals and timers are immediately pushed to the SDK as
well).
The new design would change from this
push-all-data-through-topologically-sorted-stages model to having queues
for "bundles", or for elements that are awaiting processing, and routing
them to the appropriate bundle processing subgraph.
The design is here: http://s.apache.org/streaming-fn-runner-py
I expect
I'd appreciate comments and everything : )
Best
-P.
Re: [design] A streaming Fn API runner for Python
Posted by Sam Rohde <sr...@google.com>.
Thanks for picking this up again Pablo, I wrote some small comments
concerning the TestStream.
On Tue, Oct 15, 2019 at 4:42 PM Robert Bradshaw <ro...@google.com> wrote:
> Very excited to see this! I've added some comments to the doc.
>
> On Tue, Oct 15, 2019 at 3:43 PM Pablo Estrada <pa...@google.com> wrote:
>
>> I've just been informed that access wasn't open. I've since opened access
>> to it.
>> Thanks
>> -P.
>>
>> On Tue, Oct 15, 2019 at 2:10 PM Pablo Estrada <pa...@google.com> wrote:
>>
>>> Hello all,
>>> I am planning to work on removing the old BundleBasedDirectRunner, and
>>> expand the FnApiRunner to work on streaming as well as batch.
>>> Currently, the FnApiRunner orders the processing graph topologically,
>>> and "pushes" all the data through each stage in topological order (deferred
>>> inputs such as residuals and timers are immediately pushed to the SDK as
>>> well).
>>> The new design would change from this
>>> push-all-data-through-topologically-sorted-stages model to having queues
>>> for "bundles", or for elements that are awaiting processing, and routing
>>> them to the appropriate bundle processing subgraph.
>>>
>>> The design is here: http://s.apache.org/streaming-fn-runner-py
>>>
>>> I expect
>>>
>>> I'd appreciate comments and everything : )
>>> Best
>>> -P.
>>>
>>
Re: [design] A streaming Fn API runner for Python
Posted by Robert Bradshaw <ro...@google.com>.
Very excited to see this! I've added some comments to the doc.
On Tue, Oct 15, 2019 at 3:43 PM Pablo Estrada <pa...@google.com> wrote:
> I've just been informed that access wasn't open. I've since opened access
> to it.
> Thanks
> -P.
>
> On Tue, Oct 15, 2019 at 2:10 PM Pablo Estrada <pa...@google.com> wrote:
>
>> Hello all,
>> I am planning to work on removing the old BundleBasedDirectRunner, and
>> expand the FnApiRunner to work on streaming as well as batch.
>> Currently, the FnApiRunner orders the processing graph topologically, and
>> "pushes" all the data through each stage in topological order (deferred
>> inputs such as residuals and timers are immediately pushed to the SDK as
>> well).
>> The new design would change from this
>> push-all-data-through-topologically-sorted-stages model to having queues
>> for "bundles", or for elements that are awaiting processing, and routing
>> them to the appropriate bundle processing subgraph.
>>
>> The design is here: http://s.apache.org/streaming-fn-runner-py
>>
>> I expect
>>
>> I'd appreciate comments and everything : )
>> Best
>> -P.
>>
>
Re: [design] A streaming Fn API runner for Python
Posted by Pablo Estrada <pa...@google.com>.
I've just been informed that access wasn't open. I've since opened access
to it.
Thanks
-P.
On Tue, Oct 15, 2019 at 2:10 PM Pablo Estrada <pa...@google.com> wrote:
> Hello all,
> I am planning to work on removing the old BundleBasedDirectRunner, and
> expand the FnApiRunner to work on streaming as well as batch.
> Currently, the FnApiRunner orders the processing graph topologically, and
> "pushes" all the data through each stage in topological order (deferred
> inputs such as residuals and timers are immediately pushed to the SDK as
> well).
> The new design would change from this
> push-all-data-through-topologically-sorted-stages model to having queues
> for "bundles", or for elements that are awaiting processing, and routing
> them to the appropriate bundle processing subgraph.
>
> The design is here: http://s.apache.org/streaming-fn-runner-py
>
> I expect
>
> I'd appreciate comments and everything : )
> Best
> -P.
>