[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground

["Konstantin Knauf (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882170#comment-16882170 ] 

Konstantin Knauf commented on FLINK-12749:
------------------------------------------

In order to spin up the playground the users need a few files on their machine, namely the following:
{noformat}
playground/
├── docker-compose.yaml
└── conf
    └── flink-conf.yaml
    └── log4j-cli.properties
    └── log4j-console.properties
{noformat}
The conf directory will eventually not be necessary anymore, while the docker-compose configuration is of course a prerequisite. I see two options to publish these configurations files:

*Option 1: Embedded in the Documentation*

The necessary files can be part of the documentation (expandable) and new users create the directory structure by copying the content from there.

Pro:
 * Correct Flink Version is automatically populated.
 * Playground is only maintained at one place, not additional dependencies (besides `library/flink` image and `StateMachineExample`)

Contra:
 * cumbersome and error-prone for the user

*Option 2: Dedicated Repository for Playground Setup*

We create a new Apache Flink repository `apache/flink-playgrounds` with the following directory structure:
{noformat}
flink-playground
└──flink-cluster-playground/
       ├── docker-compose.yaml
       └── conf
           └── flink-conf.yaml
           └── log4j-cli.properties
           └── log4j-console.properties
└──interactive-sql-playground/
{noformat}

The `interactive-sql-playground` will contain whatever is neccessry for FLINK-12750. 

We could have tags for each Flink version, so that we can automatically link to the correct tag in the `flink-playgrounds` repository from the respective documentation version.

Pro:
 * user only needs to checkout this small repository and is ready to go.
 * we can make sure the playground setup always works, because we explicitly bump the Flink version once the new images are ready and we have tested the playground

Contra:
 * additional repository to maintain next to the documentation

Any opinions? [~fhueske] [~rmetzger] [~sjwiesman]

> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---------------------------------------------------------------
>
>                 Key: FLINK-12749
>                 URL: https://issues.apache.org/jira/browse/FLINK-12749
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Documentation
>            Reporter: Konstantin Knauf
>            Assignee: Konstantin Knauf
>            Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), including a step-by-step guide for some common commands (job submission, savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be the most maintainable, I think. It would be preferable, if we don't need to host any custom images for this, but can rely on the existing plain Flink images.
> * Which Flink jobs to use? An updated version {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} might be a good option as it can with or without Kafka and contains a data generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ seems to be well-maintained and is openly available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)