Iteration feature merged to devel-3

[David Yan <da...@datatorrent.com>]

Hi all,

The iteration feature has been merged to the devel-3 branch in Apex Core.

We introduced a special kind of operator called DelayOperator, which takes
incoming windows and increments the window ID by one for all outgoing
ports, thus allowing loops in the topology.

This feature will be included with Apex Core 3.3.0 release.

*Please help test this feature by checking out the latest devel-3 branch in
Apex Core and building your own iteration topology.*

You can take a look at this pull request in Apex Malhar (not merged yet)
for an example of the usage of the iteration feature:

https://github.com/apache/incubator-apex-malhar/pull/177

This iteration feature opens up Apex to the world of iterative streaming
algorithms, and will be especially useful for ML training algorithms.

Known issues:

- Input Operator: The topology still requires at least one regular
InputOperator to generate the windows, even though in some cases,
technically an InputOperator is not needed because the DelayOperator is
capable of creating data for the first window.
- Locality: for now THREAD_LOCAL is not allowed in the iteration loop,
though it should be technically possible.

Coming soon:

- DelayByNOperator: A DelayOperator that simulates +N increment on top of
the native +1 increment by the engine.

Thank you!

David