[jira] [Created] (FLINK-8971) Create general purpose testing job

["Till Rohrmann (JIRA)" <ji...@apache.org>]

Till Rohrmann created FLINK-8971:
------------------------------------

             Summary: Create general purpose testing job
                 Key: FLINK-8971
                 URL: https://issues.apache.org/jira/browse/FLINK-8971
             Project: Flink
          Issue Type: Task
          Components: Tests
    Affects Versions: 1.5.0
            Reporter: Till Rohrmann
             Fix For: 1.5.0


In order to write better end-to-end tests we need a general purpose testing job which comprises as many Flink aspects as possible. These include different types for records and state, user defined components, state types and operators.

The job should allow to activate a certain misbehavior, such as slowing certain paths down or throwing exceptions to simulate failures.

The job should come with a data generator which generates input data such that the job can verify it's own behavior. This includes the state as well as the input/output records.

We already have the [heavily misbehaved job|https://github.com/rmetzger/flink-testing-jobs/blob/flink-1.4/basic-jobs/src/main/java/com/dataartisans/heavymisbehaved/HeavyMisbehavedJob.java] which simulates some misbehavior. There is also the [state machine job|https://github.com/dataArtisans/flink-testing-jobs/tree/master/streaming-state-machine] which can verify itself for invalid state changes which indicate data loss. We should incorporate their characteristics into the new general purpose job.

Additionally, the general purpose job should contain the following aspects:
* Job containing a sliding window aggregation
* At least one generic Kryo type
* At least one generic Avro type
* At least one Avro specific record type
* At least one input type for which we register a Kryo serializer
* At least one input type for which we provide a user defined serializer
* At least one state type for which we provide a user defined serializer
* At least one state type which uses the AvroSerializer
* Include an operator with ValueState
* Value state changes should be verified (e.g. predictable series of values)
* Include an operator with operator state
* Include an operator with broadcast state
* Broadcast state changes should be verified (e.g. predictable series of values)
* Include union state
* User defined watermark assigner

This issue is intended to serve as an umbrella issue for developing and extending this job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)