Design Doc for Interactive DataFrame Operations in Beam

[Andy Ye <ye...@google.com>]

Hi everyone,

I've written up a design doc
<https://s.apache.org/interactive-dataframe-operations> [1] on enabling
interactive DataFrame operations – for plotting operations, and operations
that produce non-deferred results – that eagerly trigger execution in Beam.
I'd appreciate any feedback. Thanks!

Summary:
The Beam DataFrame API operates in a deferred manner. The pandas library,
on the other hand, executes eagerly on in-memory datasets, making it useful
in interactive contexts. Some operations that make sense in pandas’
eagerly-executed context don’t make sense in Beam’s deferred context. An
approach to this problem is with an “execution-on-demand” model, in which
some operations implicitly run the pipeline, collect the dataset (behind
the scenes), and then yield the result.

Best,
Andy

[1] https://s.apache.org/interactive-dataframe-operations