You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Shubham Bansal (Jira)" <ji...@apache.org> on 2022/05/10 09:26:00 UTC
[jira] [Comment Edited] (FLINK-27506) update playgrounds for Flink 1.14
[ https://issues.apache.org/jira/browse/FLINK-27506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534258#comment-17534258 ]
Shubham Bansal edited comment on FLINK-27506 at 5/10/22 9:25 AM:
-----------------------------------------------------------------
[~danderson], thanks for the help.
I looked at the [commit|https://github.com/shba24/flink-playgrounds/commit/a55ff0a5b3e324d3cc77e405892960cac03efb7a] for FLINK-23099 and here are my findings for the updating table-walkthrough playground for Flink 1.14 after testing.
1. For the code below in SpendReport.java
{code:java}
tEnv.executeSql("CREATE TABLE transactions (\n" +
" account_id BIGINT,\n" +
" amount BIGINT,\n" +
" transaction_time TIMESTAMP(3),\n" +
" WATERMARK FOR transaction_time AS transaction_time - INTERVAL '5' SECOND\n" +
") WITH (\n" +
" 'connector' = 'kafka',\n" +
" 'topic' = 'transactions',\n" +
" 'properties.bootstrap.servers' = 'kafka:9092',\n" +
" 'format' = 'csv'\n" +
")");
{code}
we need to add
{code:java}
'scan.startup.mode' = 'earliest-offset',
{code}
in the properties, otherwise `Kafka-connector` defaults to the `group-offsets` for the `scan.startup.mode` which will trigger FLINK-24277 causing an exception at runtime, as only `earliest-offset` and `latest-offset` are valid startup modes when we are not using any group id on the consumer. This seems like a recent change after 1.13 was released last year.
2. Docker image for maven:3.6-jdk-8-slim is not available anymore. So, I increased the version to maven:3.8-jdk-8-slim which is working just fine.
3. Another issue I am seeing with `wurstmeister/kafka:2.12-2.2.1` is that it is getting stuck with "waiting for Kafka to be ready". So I tried with the latest version of the docker image `wurstmeister/kafka:2.13-2.8.1` which is working. Also, I need to remove the following from the docker-compose.yml file
{code:java}
volumes:
- /var/run/docker.sock:/var/run/docker.sock {code}
as mentioned in the GitHub issue [here.|https://github.com/wurstmeister/kafka-docker/issues/591]
Otherwise, it's getting stuck with the same error during the startup.
4. Another change I had to make was related to `flink-table-planner-blink` which has been discontinued as a separate package. So, I had to remove the following from the pom.xml to make it work.
{code:java}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner-blink_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
</dependency> {code}
Environment: I tested this on my macOS with arm64/v8 with java8.
Result: I have the diff ready to raise the PR if you are fine with the above explanation.
Let me know.
Thanks,
Shubham
was (Author: JIRAUSER289179):
[~danderson], thanks for the help.
I looked at the [commit|https://github.com/shba24/flink-playgrounds/commit/a55ff0a5b3e324d3cc77e405892960cac03efb7a] for FLINK-23099 and here are my findings for the updating table-walkthrough playground for Flink 1.14 after testing.
1. For the code below in SpendReport.java
{code:java}
tEnv.executeSql("CREATE TABLE transactions (\n" +
" account_id BIGINT,\n" +
" amount BIGINT,\n" +
" transaction_time TIMESTAMP(3),\n" +
" WATERMARK FOR transaction_time AS transaction_time - INTERVAL '5' SECOND\n" +
") WITH (\n" +
" 'connector' = 'kafka',\n" +
" 'topic' = 'transactions',\n" +
" 'properties.bootstrap.servers' = 'kafka:9092',\n" +
" 'format' = 'csv'\n" +
")");
{code}
we need to add
{code:java}
'scan.startup.mode' = 'earliest-offset',
{code}
in the properties, otherwise `Kafka-connector` defaults to the `group-offsets` for the `scan.startup.mode` which will trigger FLINK-24277 causing an exception at runtime, as only `earliest-offset` and `latest-offset` are valid startup modes when we are not using any group id on the consumer. This seems like a recent change after 1.13 was released last year.
2. Docker image for maven:3.6-jdk-8-slim is not available anymore. So, I increased the version to maven:3.8-jdk-8-slim which is working just fine.
3. Another issue I am seeing with `wurstmeister/kafka:2.12-2.2.1` is that it is getting stuck with "waiting for Kafka to be ready". So I tried with the latest version of the docker image `wurstmeister/kafka:2.13-2.8.1` which is working. Also, I need to remove the following from the docker-compose.yml file
{code:java}
volumes:
- /var/run/docker.sock:/var/run/docker.sock {code}
as mentioned in the GitHub issue [here.|https://github.com/wurstmeister/kafka-docker/issues/591]
Otherwise, it's getting stuck with the same error during the startup.
Environment: I tested this on my macOS with arm64/v8 with java8.
Result: I have the diff ready to raise the PR if you are fine with the above explanation.
Let me know.
Thanks,
Shubham
> update playgrounds for Flink 1.14
> ---------------------------------
>
> Key: FLINK-27506
> URL: https://issues.apache.org/jira/browse/FLINK-27506
> Project: Flink
> Issue Type: Improvement
> Components: Documentation / Training / Exercises
> Affects Versions: 1.14.4
> Reporter: David Anderson
> Priority: Major
> Labels: starter
>
> All of the flink-playgrounds need to be updated for 1.14.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)