Distributed Operator Update

[Timothy Farkas <ti...@datatorrent.com>]

Hi All,

Sandesh and I have been working on
https://malhar.atlassian.net/browse/APEX-119 which aims to bring true
distributed data structures to Apex. We have proposed a way to implement
distributed data structures natively within Apache Apex. We have finished
an analysis of what it would take to use other (non native) distributed
data structures to accomplish the same task.

It turns out that using a distributed data structure fault tolerantly (like
those provided by Apache Ignite) when it is not embedded into an Apex
application is very difficult. It requires extra state and logic to manage
windowing in a system outside of Apex, It also requires partitions reading
from that data structure to communicate with one another anyway. The full
analysis is here:

https://docs.google.com/document/d/1Rau76WxAycyN9vQqP2bqDWZAwLw0u23xSh0_5fQ1980/edit?usp=sharing

So the good news is if we want distributed data structures to be
efficiently used in an Apex application, we'll need to implement
distributed operators :)

Thanks,
Tim