You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/07/05 12:11:12 UTC
[GitHub] [flink] MartijnVisser opened a new pull request, #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.1.2
MartijnVisser opened a new pull request, #20170:
URL: https://github.com/apache/flink/pull/20170
## What is the purpose of the change
We have updated the used Kafka Clients to v3.1.1 via FLINK-28060, but we are using Confluent Platform 6.2.2 which supports up to Kafka 2.8.0.
We should update to Confluent Platform v7.1.2, which includes support for Kafka 3.1.0.
## Brief change log
* Updated all references to Confluent Platform versions (6.2.2) to 7.1.2
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1269244849
@zentol Seems I've finally fixed this so would be great if you could have a final look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r990173189
##########
tools/azure-pipelines/cache_docker_images.sh:
##########
@@ -28,7 +28,7 @@ then
fi
# This is the pattern that determines which containers we save.
-DOCKER_IMAGE_CACHE_PATTERN="testcontainers|kafka|postgres|mysql|pulsar|cassandra"
+DOCKER_IMAGE_CACHE_PATTERN="testcontainers|kafka|postgres|mysql|pulsar|cassandra|schema_registry"
Review Comment:
you sure this doesn't need a `-` instead of `_`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.1.3
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1202600471
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser merged pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser merged PR #20170:
URL: https://github.com/apache/flink/pull/20170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.1.3
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1202678582
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r989997402
##########
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java:
##########
@@ -28,7 +28,9 @@
*/
public class DockerImageVersions {
- public static final String KAFKA = "confluentinc/cp-kafka:6.2.2";
+ public static final String KAFKA = "confluentinc/cp-kafka:7.2.2";
+
+ public static final String SCHEMA_REGISTRY = "confluentinc/cp-schema-registry:7.2.2";
Review Comment:
I don't remember. I think because in some cases we pull LATEST, so we'd invalidate the cache quite frequently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.1.2
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1174998197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81ad320e86b03d24aa173fd835a4ed658bb535b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81ad320e86b03d24aa173fd835a4ed658bb535b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81ad320e86b03d24aa173fd835a4ed658bb535b8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.1.3
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1202544064
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1271020448
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r989429992
##########
flink-connectors/flink-connector-kafka/pom.xml:
##########
@@ -111,11 +111,12 @@ under the License.
</dependency>
<!-- Required to execute the kafka server for testing. Please change the zookeeper version accordingly when changing the Kafka version
+ Currently synced with Kafka 3.2.1
https://github.com/apache/kafka/blob/839b886f9b732b151e1faeace7303c80641c08c4/gradle/dependencies.gradle#L122 -->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
- <version>3.5.9</version>
+ <version>3.6.3</version>
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1273048877
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1271021662
I've ran the failing test multiple times locally, passing every time. Looks like [FLINK-24119](https://issues.apache.org/jira/browse/FLINK-24119) but that has been resolved recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r990199370
##########
tools/azure-pipelines/cache_docker_images.sh:
##########
@@ -28,7 +28,7 @@ then
fi
# This is the pattern that determines which containers we save.
-DOCKER_IMAGE_CACHE_PATTERN="testcontainers|kafka|postgres|mysql|pulsar|cassandra"
+DOCKER_IMAGE_CACHE_PATTERN="testcontainers|kafka|postgres|mysql|pulsar|cassandra|schema_registry"
Review Comment:
FFS. Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1272611849
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r989929080
##########
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java:
##########
@@ -28,7 +28,9 @@
*/
public class DockerImageVersions {
- public static final String KAFKA = "confluentinc/cp-kafka:6.2.2";
+ public static final String KAFKA = "confluentinc/cp-kafka:7.2.2";
+
+ public static final String SCHEMA_REGISTRY = "confluentinc/cp-schema-registry:7.2.2";
Review Comment:
also needs to be added in `tools/azure-pipelines/cache_docker_images.sh`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r989976980
##########
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java:
##########
@@ -28,7 +28,9 @@
*/
public class DockerImageVersions {
- public static final String KAFKA = "confluentinc/cp-kafka:6.2.2";
+ public static final String KAFKA = "confluentinc/cp-kafka:7.2.2";
+
+ public static final String SCHEMA_REGISTRY = "confluentinc/cp-schema-registry:7.2.2";
Review Comment:
I'll add it. Any reason why we don't cache all images? Like RabbitMQ, Kinesalite, Minio etc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1276309598
@zentol Can I get one more look at this one too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r988947010
##########
flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/pom.xml:
##########
@@ -156,6 +156,11 @@ under the License.
<version>${project.version}</version>
<type>test-jar</type>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>30.1.1-jre</version>
+ </dependency>
Review Comment:
So who actually needs this?
##########
flink-connectors/flink-connector-kafka/pom.xml:
##########
@@ -111,11 +111,12 @@ under the License.
</dependency>
<!-- Required to execute the kafka server for testing. Please change the zookeeper version accordingly when changing the Kafka version
+ Currently synced with Kafka 3.2.1
https://github.com/apache/kafka/blob/839b886f9b732b151e1faeace7303c80641c08c4/gradle/dependencies.gradle#L122 -->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
- <version>3.5.9</version>
+ <version>3.6.3</version>
Review Comment:
Ideally this sticks with `${zookeeper.version}` if possible.
##########
flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/SQLClientSchemaRegistryITCase.java:
##########
@@ -84,7 +86,7 @@ public class SQLClientSchemaRegistryITCase {
@ClassRule
public static final SchemaRegistryContainer REGISTRY =
- new SchemaRegistryContainer("6.2.2")
+ new SchemaRegistryContainer("7.2.2")
Review Comment:
Can we change this constructor to accept a full image name, and add it to DockerImageVersions for caching purposes?
##########
flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/SQLClientSchemaRegistryITCase.java:
##########
@@ -71,6 +71,8 @@ public class SQLClientSchemaRegistryITCase {
private static final Path sqlToolBoxJar = ResourceTestUtils.getResource(".*SqlToolbox.jar");
private final Path sqlConnectorKafkaJar = ResourceTestUtils.getResource(".*kafka.jar");
+ private final Path sqlGuavaJar = ResourceTestUtils.getResource(".*guava.jar");
Review Comment:
```suggestion
private final Path guavaJar = ResourceTestUtils.getResource(".*guava.jar");
```
This isn't a sql jar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r989430227
##########
flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/pom.xml:
##########
@@ -156,6 +156,11 @@ under the License.
<version>${project.version}</version>
<type>test-jar</type>
</dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>30.1.1-jre</version>
+ </dependency>
Review Comment:
The Schema Registry service
##########
flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/SQLClientSchemaRegistryITCase.java:
##########
@@ -71,6 +71,8 @@ public class SQLClientSchemaRegistryITCase {
private static final Path sqlToolBoxJar = ResourceTestUtils.getResource(".*SqlToolbox.jar");
private final Path sqlConnectorKafkaJar = ResourceTestUtils.getResource(".*kafka.jar");
+ private final Path sqlGuavaJar = ResourceTestUtils.getResource(".*guava.jar");
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on a diff in pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #20170:
URL: https://github.com/apache/flink/pull/20170#discussion_r989430410
##########
flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/SQLClientSchemaRegistryITCase.java:
##########
@@ -84,7 +86,7 @@ public class SQLClientSchemaRegistryITCase {
@ClassRule
public static final SchemaRegistryContainer REGISTRY =
- new SchemaRegistryContainer("6.2.2")
+ new SchemaRegistryContainer("7.2.2")
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #20170: [FLINK-28405][Connector/Kafka] Update Confluent Platform images used for testing to v7.2.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #20170:
URL: https://github.com/apache/flink/pull/20170#issuecomment-1270926463
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org