Testing iterative data flows

[Ken Krugler <kk...@transpac.com>]

Hi all,

What’s the recommended way currently to test a streaming data flow that has iterations?

I know that using timeouts in tests (which FLINK-2390 <https://issues.apache.org/jira/browse/FLINK-2390> also discusses) isn’t reliable, and it’s hard to know when a job with iterations is really “done” in the context of a test.

Are there any other approaches with current versions of Flink that would be better than an arbitrary timeout?

Thanks,

— Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

Re: Testing iterative data flows

[Ufuk Celebi <uc...@apache.org>]

Hey Ken! Unfortunately, no. But Paris just posted a proposal to
improve this: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-13-Consistent-Processing-with-Loops-tt14149.html

On Wed, Oct 26, 2016 at 11:10 PM, Ken Krugler
<kk...@transpac.com> wrote:
> Hi all,
>
> What’s the recommended way currently to test a streaming data flow that has
> iterations?
>
> I know that using timeouts in tests (which FLINK-2390 also discusses) isn’t
> reliable, and it’s hard to know when a job with iterations is really “done”
> in the context of a test.
>
> Are there any other approaches with current versions of Flink that would be
> better than an arbitrary timeout?
>
> Thanks,
>
> — Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>