You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ma...@apache.org on 2023/02/03 09:25:42 UTC
[flink-connector-gcp-pubsub] branch main updated (c546198 -> f163009)
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
from c546198 [FLINK-30058][Connector/Google PubSub] Add release tools as submodule
new 9e02c4c [FLINK-9311] [pubsub] Clean up / add documentation and style issues in the PubSub connector
new 6c5cd55 [FLINK-9311] [pubsub] Improvements to builders + minor improvement to PubSubSink flush logic
new 1512bc1 Update version to 1.10-SNAPSHOT
new 6f92e4f [FLINK-13231] [pubsub] Replace Max outstanding acknowledgement ids limit with a FlinkConnectorRateLimiter
new c93e10d Update version to 1.11-SNAPSHOT
new 55f84bd Update version to 1.12-SNAPSHOT
new 62d4279 [FLINK-18607][build] Give the maven module a human readable name
new 3cdd0ac [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
new eaad434 Update version to 1.13-SNAPSHOT
new 977cff2 [FLINK-20651] Format code with Spotless/google-java-format
new fcf58ac Update version to 1.14-SNAPSHOT
new 1c9423f Update version to 1.15-SNAPSHOT
new 2ae4629 [FLINK-24018][build] Remove Scala dependencies from Java APIs
new 83174ac [FLINK-25039][build] Disable shading of test jar by default
new 824d973 Update version to 1.16-SNAPSHOT
new 942a616 [FLINK-24721][build] Remove unnecessary relativePath to parent
new ca0dd1d Update version to 1.17-SNAPSHOT
new 7d8090b [FLINK-30872][Connector/Google PubSub] Move GCP example to GCP connector repository
new f163009 [hotfix] Change Flink binary URL to ASF archive
The 19 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/push_pr.yml | 2 +-
flink-connector-gcp-pubsub/pom.xml | 17 ----
flink-examples-streaming-gcp-pubsub/pom.xml | 110 +++++++++++++++++++++
.../examples/gcp/pubsub/IntegerSerializer.java | 55 +++++++++++
.../examples/gcp/pubsub/PubSubExample.java | 90 +++++++++++++++++
.../examples/gcp/pubsub/PubSubPublisher.java | 66 +++++++++++++
pom.xml | 11 +++
7 files changed, 333 insertions(+), 18 deletions(-)
create mode 100644 flink-examples-streaming-gcp-pubsub/pom.xml
create mode 100644 flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/IntegerSerializer.java
create mode 100644 flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
create mode 100644 flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
[flink-connector-gcp-pubsub] 09/19: Update version to 1.13-SNAPSHOT
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit eaad434fa0502a1446ee402003f518f66581e6a4
Author: Dian Fu <di...@apache.org>
AuthorDate: Thu Nov 26 14:20:51 2020 +0800
Update version to 1.13-SNAPSHOT
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index 6378010..7e7e2b9 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -24,7 +24,7 @@ under the License.
<parent>
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.12-SNAPSHOT</version>
+ <version>1.13-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
[flink-connector-gcp-pubsub] 03/19: Update version to 1.10-SNAPSHOT
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 1512bc127d46a65b74d6f9269f2b5ea06e06d270
Author: Kurt Young <yk...@gmail.com>
AuthorDate: Fri Jul 12 14:25:23 2019 +0800
Update version to 1.10-SNAPSHOT
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index ab1c91f..969f153 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -24,7 +24,7 @@ under the License.
<parent>
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.9-SNAPSHOT</version>
+ <version>1.10-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
[flink-connector-gcp-pubsub] 01/19: [FLINK-9311] [pubsub] Clean up / add documentation and style issues in the PubSub connector
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 9e02c4c71bde038f4f968e90620deb4c9bd2df38
Author: Richard Deurwaarder <rd...@bol.com>
AuthorDate: Sat Aug 25 15:21:54 2018 +0200
[FLINK-9311] [pubsub] Clean up / add documentation and style issues in the PubSub connector
---
flink-examples-streaming-gcp-pubsub/pom.xml | 108 +++++++++++++++++++++
.../examples/gcp/pubsub/IntegerSerializer.java | 54 +++++++++++
.../examples/gcp/pubsub/PubSubExample.java | 84 ++++++++++++++++
.../examples/gcp/pubsub/PubSubPublisher.java | 67 +++++++++++++
4 files changed, 313 insertions(+)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
new file mode 100644
index 0000000..ab1c91f
--- /dev/null
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -0,0 +1,108 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <artifactId>flink-examples-build-helper</artifactId>
+ <groupId>org.apache.flink</groupId>
+ <version>1.9-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <artifactId>flink-examples-streaming-gcp-pubsub_${scala.binary.version}</artifactId>
+ <name>flink-examples-streaming-gcp-pubsub</name>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-gcp-pubsub_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-examples-streaming_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <finalName>PubSub</finalName>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>shade-flink</id>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <shadeTestJar>false</shadeTestJar>
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <createDependencyReducedPom>false</createDependencyReducedPom>
+ <transformers>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+ <mainClass>org.apache.flink.streaming.examples.gcp.pubsub.PubSubExample</mainClass>
+ </transformer>
+ </transformers>
+ <artifactSet>
+ <includes>
+ <include>org.apache.flink:flink-connector-gcp-pubsub*</include>
+ <include>com.google.cloud:google-cloud-pubsub</include>
+ <include>com.google.*:*</include>
+ <include>org.threeten:*</include>
+ <include>io.grpc:*</include>
+ <include>io.opencensus:*</include>
+ </includes>
+ </artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>com.google</pattern>
+ <shadedPattern>org.apache.flink.streaming.examples.gcp.pubsub.shaded.com.google</shadedPattern>
+ </relocation>
+ </relocations>
+
+ <filters>
+ <filter>
+ <artifact>org.apache.flink:flink-examples-streaming_*</artifact>
+ <includes>
+ <include>org/apache/flink/streaming/examples/gcp/pubsub/**</include>
+ </includes>
+ </filter>
+ </filters>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/IntegerSerializer.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/IntegerSerializer.java
new file mode 100644
index 0000000..3c1eab4
--- /dev/null
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/IntegerSerializer.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.gcp.pubsub;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.connectors.gcp.pubsub.common.PubSubDeserializationSchema;
+
+import com.google.pubsub.v1.PubsubMessage;
+
+import java.io.IOException;
+import java.math.BigInteger;
+
+/**
+ * Deserialization schema to deserialize messages produced by {@link PubSubPublisher}.
+ * The byte[] received by this schema must contain a single Integer.
+ */
+class IntegerSerializer implements PubSubDeserializationSchema<Integer>, SerializationSchema<Integer> {
+
+ @Override
+ public Integer deserialize(PubsubMessage message) throws IOException {
+ return new BigInteger(message.getData().toByteArray()).intValue();
+ }
+
+ @Override
+ public boolean isEndOfStream(Integer integer) {
+ return false;
+ }
+
+ @Override
+ public TypeInformation<Integer> getProducedType() {
+ return TypeInformation.of(Integer.class);
+ }
+
+ @Override
+ public byte[] serialize(Integer integer) {
+ return BigInteger.valueOf(integer).toByteArray();
+ }
+}
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
new file mode 100644
index 0000000..7b66577
--- /dev/null
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.gcp.pubsub;
+
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSink;
+import org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A simple PubSub example.
+ *
+ * <p>Before starting a flink job it will publish 10 messages on the input topic.
+ *
+ * Then a flink job is started to read these 10 messages from the input-subscription,
+ * it will print them to stdout
+ * and then write them to a the output-topic.</p>
+ */
+public class PubSubExample {
+ private static final Logger LOG = LoggerFactory.getLogger(PubSubExample.class);
+
+ public static void main(String[] args) throws Exception {
+ // parse input arguments
+ final ParameterTool parameterTool = ParameterTool.fromArgs(args);
+
+ if (parameterTool.getNumberOfParameters() < 3) {
+ System.out.println("Missing parameters!\n" +
+ "Usage: flink run PubSub.jar --input-subscription <subscription> --input-topicName <topic> --output-topicName <output-topic> " +
+ "--google-project <google project name> ");
+ return;
+ }
+
+ String projectName = parameterTool.getRequired("google-project");
+ String inputTopicName = parameterTool.getRequired("input-topicName");
+ String subscriptionName = parameterTool.getRequired("input-subscription");
+ String outputTopicName = parameterTool.getRequired("output-topicName");
+
+ PubSubPublisher pubSubPublisher = new PubSubPublisher(projectName, inputTopicName);
+ pubSubPublisher.publish(10);
+
+ runFlinkJob(projectName, subscriptionName, outputTopicName);
+ }
+
+ private static void runFlinkJob(String projectName, String subscriptionName, String outputTopicName) throws Exception {
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.enableCheckpointing(1000L);
+
+ env.addSource(PubSubSource.newBuilder(Integer.class)
+ .withDeserializationSchema(new IntegerSerializer())
+ .withProjectName(projectName)
+ .withSubscriptionName(subscriptionName)
+ .build())
+ .map(PubSubExample::printAndReturn).disableChaining()
+ .addSink(PubSubSink.newBuilder(Integer.class)
+ .withSerializationSchema(new IntegerSerializer())
+ .withProjectName(projectName)
+ .withTopicName(outputTopicName).build());
+
+ env.execute("Flink Streaming PubSubReader");
+ }
+
+ private static Integer printAndReturn(Integer i) {
+ LOG.info("Processed message with payload: " + i);
+ return i;
+ }
+}
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
new file mode 100644
index 0000000..8f7bfe6
--- /dev/null
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.gcp.pubsub;
+
+import com.google.cloud.pubsub.v1.Publisher;
+import com.google.protobuf.ByteString;
+import com.google.pubsub.v1.ProjectTopicName;
+import com.google.pubsub.v1.PubsubMessage;
+
+import java.math.BigInteger;
+
+/**
+ * Helper class to send PubSubMessages to a PubSub topic.
+ */
+class PubSubPublisher {
+ private final String projectName;
+ private final String topicName;
+
+ PubSubPublisher(String projectName, String topicName) {
+ this.projectName = projectName;
+ this.topicName = topicName;
+ }
+
+ /**
+ * Publish messages with as payload a single integer.
+ * The integers inside the messages start from 0 and increase by one for each message send.
+ * @param amountOfMessages amount of messages to send
+ */
+ void publish(int amountOfMessages) {
+ Publisher publisher = null;
+ try {
+ publisher = Publisher.newBuilder(ProjectTopicName.of(projectName, topicName)).build();
+ for (int i = 0; i < amountOfMessages; i++) {
+ ByteString messageData = ByteString.copyFrom(BigInteger.valueOf(i).toByteArray());
+ PubsubMessage message = PubsubMessage.newBuilder().setData(messageData).build();
+ publisher.publish(message).get();
+
+ System.out.println("Published message: " + i);
+ Thread.sleep(100L);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ } finally {
+ try {
+ if (publisher != null) {
+ publisher.shutdown();
+ }
+ } catch (Exception e) {
+ }
+ }
+ }
+}
[flink-connector-gcp-pubsub] 10/19: [FLINK-20651] Format code with Spotless/google-java-format
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 977cff2d79967b4e1392b84c180ec4e2655eceef
Author: Rufus Refactor <de...@flink.apache.org>
AuthorDate: Mon Dec 28 14:30:59 2020 +0100
[FLINK-20651] Format code with Spotless/google-java-format
---
.../examples/gcp/pubsub/IntegerSerializer.java | 47 ++++++------
.../examples/gcp/pubsub/PubSubExample.java | 87 ++++++++++++----------
.../examples/gcp/pubsub/PubSubPublisher.java | 71 +++++++++---------
3 files changed, 105 insertions(+), 100 deletions(-)
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/IntegerSerializer.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/IntegerSerializer.java
index 3c1eab4..0c5dcb1 100644
--- a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/IntegerSerializer.java
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/IntegerSerializer.java
@@ -27,28 +27,29 @@ import java.io.IOException;
import java.math.BigInteger;
/**
- * Deserialization schema to deserialize messages produced by {@link PubSubPublisher}.
- * The byte[] received by this schema must contain a single Integer.
+ * Deserialization schema to deserialize messages produced by {@link PubSubPublisher}. The byte[]
+ * received by this schema must contain a single Integer.
*/
-class IntegerSerializer implements PubSubDeserializationSchema<Integer>, SerializationSchema<Integer> {
-
- @Override
- public Integer deserialize(PubsubMessage message) throws IOException {
- return new BigInteger(message.getData().toByteArray()).intValue();
- }
-
- @Override
- public boolean isEndOfStream(Integer integer) {
- return false;
- }
-
- @Override
- public TypeInformation<Integer> getProducedType() {
- return TypeInformation.of(Integer.class);
- }
-
- @Override
- public byte[] serialize(Integer integer) {
- return BigInteger.valueOf(integer).toByteArray();
- }
+class IntegerSerializer
+ implements PubSubDeserializationSchema<Integer>, SerializationSchema<Integer> {
+
+ @Override
+ public Integer deserialize(PubsubMessage message) throws IOException {
+ return new BigInteger(message.getData().toByteArray()).intValue();
+ }
+
+ @Override
+ public boolean isEndOfStream(Integer integer) {
+ return false;
+ }
+
+ @Override
+ public TypeInformation<Integer> getProducedType() {
+ return TypeInformation.of(Integer.class);
+ }
+
+ @Override
+ public byte[] serialize(Integer integer) {
+ return BigInteger.valueOf(integer).toByteArray();
+ }
}
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
index b79c67e..b4666cf 100644
--- a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
@@ -30,56 +30,61 @@ import org.slf4j.LoggerFactory;
*
* <p>Before starting a flink job it will publish 10 messages on the input topic.
*
- * Then a flink job is started to read these 10 messages from the input-subscription,
- * it will print them to stdout
- * and then write them to a the output-topic.</p>
+ * <p>Then a flink job is started to read these 10 messages from the input-subscription, it will
+ * print them to stdout and then write them to a the output-topic.
*/
public class PubSubExample {
- private static final Logger LOG = LoggerFactory.getLogger(PubSubExample.class);
+ private static final Logger LOG = LoggerFactory.getLogger(PubSubExample.class);
- public static void main(String[] args) throws Exception {
- // parse input arguments
- final ParameterTool parameterTool = ParameterTool.fromArgs(args);
+ public static void main(String[] args) throws Exception {
+ // parse input arguments
+ final ParameterTool parameterTool = ParameterTool.fromArgs(args);
- if (parameterTool.getNumberOfParameters() < 3) {
- System.out.println("Missing parameters!\n" +
- "Usage: flink run PubSub.jar --input-subscription <subscription> --input-topicName <topic> --output-topicName <output-topic> " +
- "--google-project <google project name> ");
- return;
- }
+ if (parameterTool.getNumberOfParameters() < 3) {
+ System.out.println(
+ "Missing parameters!\n"
+ + "Usage: flink run PubSub.jar --input-subscription <subscription> --input-topicName <topic> --output-topicName <output-topic> "
+ + "--google-project <google project name> ");
+ return;
+ }
- String projectName = parameterTool.getRequired("google-project");
- String inputTopicName = parameterTool.getRequired("input-topicName");
- String subscriptionName = parameterTool.getRequired("input-subscription");
- String outputTopicName = parameterTool.getRequired("output-topicName");
+ String projectName = parameterTool.getRequired("google-project");
+ String inputTopicName = parameterTool.getRequired("input-topicName");
+ String subscriptionName = parameterTool.getRequired("input-subscription");
+ String outputTopicName = parameterTool.getRequired("output-topicName");
- PubSubPublisher pubSubPublisher = new PubSubPublisher(projectName, inputTopicName);
- pubSubPublisher.publish(10);
+ PubSubPublisher pubSubPublisher = new PubSubPublisher(projectName, inputTopicName);
+ pubSubPublisher.publish(10);
- runFlinkJob(projectName, subscriptionName, outputTopicName);
- }
+ runFlinkJob(projectName, subscriptionName, outputTopicName);
+ }
- private static void runFlinkJob(String projectName, String subscriptionName, String outputTopicName) throws Exception {
- final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
- env.enableCheckpointing(1000L);
+ private static void runFlinkJob(
+ String projectName, String subscriptionName, String outputTopicName) throws Exception {
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.enableCheckpointing(1000L);
- env.addSource(PubSubSource.newBuilder()
- .withDeserializationSchema(new IntegerSerializer())
- .withProjectName(projectName)
- .withSubscriptionName(subscriptionName)
- .withMessageRateLimit(1)
- .build())
- .map(PubSubExample::printAndReturn).disableChaining()
- .addSink(PubSubSink.newBuilder()
- .withSerializationSchema(new IntegerSerializer())
- .withProjectName(projectName)
- .withTopicName(outputTopicName).build());
+ env.addSource(
+ PubSubSource.newBuilder()
+ .withDeserializationSchema(new IntegerSerializer())
+ .withProjectName(projectName)
+ .withSubscriptionName(subscriptionName)
+ .withMessageRateLimit(1)
+ .build())
+ .map(PubSubExample::printAndReturn)
+ .disableChaining()
+ .addSink(
+ PubSubSink.newBuilder()
+ .withSerializationSchema(new IntegerSerializer())
+ .withProjectName(projectName)
+ .withTopicName(outputTopicName)
+ .build());
- env.execute("Flink Streaming PubSubReader");
- }
+ env.execute("Flink Streaming PubSubReader");
+ }
- private static Integer printAndReturn(Integer i) {
- LOG.info("Processed message with payload: " + i);
- return i;
- }
+ private static Integer printAndReturn(Integer i) {
+ LOG.info("Processed message with payload: " + i);
+ return i;
+ }
}
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
index ef9ad83..4569c21 100644
--- a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
@@ -24,44 +24,43 @@ import com.google.pubsub.v1.TopicName;
import java.math.BigInteger;
-/**
- * Helper class to send PubSubMessages to a PubSub topic.
- */
+/** Helper class to send PubSubMessages to a PubSub topic. */
class PubSubPublisher {
- private final String projectName;
- private final String topicName;
+ private final String projectName;
+ private final String topicName;
- PubSubPublisher(String projectName, String topicName) {
- this.projectName = projectName;
- this.topicName = topicName;
- }
+ PubSubPublisher(String projectName, String topicName) {
+ this.projectName = projectName;
+ this.topicName = topicName;
+ }
- /**
- * Publish messages with as payload a single integer.
- * The integers inside the messages start from 0 and increase by one for each message send.
- * @param amountOfMessages amount of messages to send
- */
- void publish(int amountOfMessages) {
- Publisher publisher = null;
- try {
- publisher = Publisher.newBuilder(TopicName.of(projectName, topicName)).build();
- for (int i = 0; i < amountOfMessages; i++) {
- ByteString messageData = ByteString.copyFrom(BigInteger.valueOf(i).toByteArray());
- PubsubMessage message = PubsubMessage.newBuilder().setData(messageData).build();
- publisher.publish(message).get();
+ /**
+ * Publish messages with as payload a single integer. The integers inside the messages start
+ * from 0 and increase by one for each message send.
+ *
+ * @param amountOfMessages amount of messages to send
+ */
+ void publish(int amountOfMessages) {
+ Publisher publisher = null;
+ try {
+ publisher = Publisher.newBuilder(TopicName.of(projectName, topicName)).build();
+ for (int i = 0; i < amountOfMessages; i++) {
+ ByteString messageData = ByteString.copyFrom(BigInteger.valueOf(i).toByteArray());
+ PubsubMessage message = PubsubMessage.newBuilder().setData(messageData).build();
+ publisher.publish(message).get();
- System.out.println("Published message: " + i);
- Thread.sleep(100L);
- }
- } catch (Exception e) {
- throw new RuntimeException(e);
- } finally {
- try {
- if (publisher != null) {
- publisher.shutdown();
- }
- } catch (Exception e) {
- }
- }
- }
+ System.out.println("Published message: " + i);
+ Thread.sleep(100L);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ } finally {
+ try {
+ if (publisher != null) {
+ publisher.shutdown();
+ }
+ } catch (Exception e) {
+ }
+ }
+ }
}
[flink-connector-gcp-pubsub] 17/19: Update version to 1.17-SNAPSHOT
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit ca0dd1d1da2ae0bb3596b1501a4488874158d994
Author: huangxingbo <hx...@apache.org>
AuthorDate: Mon Sep 5 10:43:15 2022 +0800
Update version to 1.17-SNAPSHOT
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index b22c935..8720ed9 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -24,7 +24,7 @@ under the License.
<parent>
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.16-SNAPSHOT</version>
+ <version>1.17-SNAPSHOT</version>
</parent>
<artifactId>flink-examples-streaming-gcp-pubsub_${scala.binary.version}</artifactId>
[flink-connector-gcp-pubsub] 19/19: [hotfix] Change Flink binary URL to ASF archive
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit f163009e3c571fce306909ff8bc46e06ee4b19a3
Author: Martijn Visser <mv...@confluent.io>
AuthorDate: Thu Feb 2 13:06:16 2023 +0100
[hotfix] Change Flink binary URL to ASF archive
---
.github/workflows/push_pr.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/push_pr.yml b/.github/workflows/push_pr.yml
index e5d6b2a..a48478a 100644
--- a/.github/workflows/push_pr.yml
+++ b/.github/workflows/push_pr.yml
@@ -26,5 +26,5 @@ jobs:
uses: apache/flink-connector-shared-utils/.github/workflows/ci.yml@ci_utils
with:
flink_version: 1.16.0
- flink_url: https://dist.apache.org/repos/dist/release/flink/flink-1.16.0/flink-1.16.0-bin-scala_2.12.tgz
+ flink_url: https://archive.apache.org/dist/flink/flink-1.16.0/flink-1.16.0-bin-scala_2.12.tgz
cache_flink_binary: true
[flink-connector-gcp-pubsub] 04/19: [FLINK-13231] [pubsub] Replace Max outstanding acknowledgement ids limit with a FlinkConnectorRateLimiter
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 6f92e4f7c455c90e60e3014a0caf126f84664cce
Author: Richard Deurwaarder <rd...@bol.com>
AuthorDate: Wed Aug 7 21:21:03 2019 +0200
[FLINK-13231] [pubsub] Replace Max outstanding acknowledgement ids limit with a FlinkConnectorRateLimiter
---
.../org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java | 1 +
1 file changed, 1 insertion(+)
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
index a960176..b79c67e 100644
--- a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
@@ -67,6 +67,7 @@ public class PubSubExample {
.withDeserializationSchema(new IntegerSerializer())
.withProjectName(projectName)
.withSubscriptionName(subscriptionName)
+ .withMessageRateLimit(1)
.build())
.map(PubSubExample::printAndReturn).disableChaining()
.addSink(PubSubSink.newBuilder()
[flink-connector-gcp-pubsub] 15/19: Update version to 1.16-SNAPSHOT
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 824d973e8a7f40bb9348d114e63d11176eae492d
Author: Yun Gao <ga...@gmail.com>
AuthorDate: Tue Mar 15 01:27:55 2022 +0800
Update version to 1.16-SNAPSHOT
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index cf5824c..ae32196 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -24,7 +24,7 @@ under the License.
<parent>
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.15-SNAPSHOT</version>
+ <version>1.16-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
[flink-connector-gcp-pubsub] 11/19: Update version to 1.14-SNAPSHOT
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit fcf58acbe1588eddc1c47b9a2c056b2519b08e15
Author: GuoWei Ma <gu...@gmail.com>
AuthorDate: Sun Apr 18 20:19:52 2021 +0800
Update version to 1.14-SNAPSHOT
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index 7e7e2b9..d582b63 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -24,7 +24,7 @@ under the License.
<parent>
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.13-SNAPSHOT</version>
+ <version>1.14-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
[flink-connector-gcp-pubsub] 13/19: [FLINK-24018][build] Remove Scala dependencies from Java APIs
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 2ae4629ffbfbcd098e6932f6319039bbdcd9ba8f
Author: Chesnay Schepler <ch...@apache.org>
AuthorDate: Tue Oct 26 00:48:06 2021 +0200
[FLINK-24018][build] Remove Scala dependencies from Java APIs
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index b5bcccc..2ff1b85 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -35,7 +35,7 @@ under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-gcp-pubsub_${scala.binary.version}</artifactId>
+ <artifactId>flink-connector-gcp-pubsub</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
[flink-connector-gcp-pubsub] 14/19: [FLINK-25039][build] Disable shading of test jar by default
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 83174ac063be34e3a1d615ac5111ca9e496a82ff
Author: Chesnay Schepler <ch...@apache.org>
AuthorDate: Thu Nov 25 11:55:32 2021 +0100
[FLINK-25039][build] Disable shading of test jar by default
---
flink-examples-streaming-gcp-pubsub/pom.xml | 1 -
1 file changed, 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index 2ff1b85..cf5824c 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -66,7 +66,6 @@ under the License.
<goal>shade</goal>
</goals>
<configuration>
- <shadeTestJar>false</shadeTestJar>
<shadedArtifactAttached>false</shadedArtifactAttached>
<createDependencyReducedPom>false</createDependencyReducedPom>
<transformers>
[flink-connector-gcp-pubsub] 02/19: [FLINK-9311] [pubsub] Improvements to builders + minor improvement to PubSubSink flush logic
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 6c5cd555525a0d9e663b5b88e12ed594520c9e66
Author: Jiangjie (Becket) Qin <be...@gmail.com>
AuthorDate: Fri Jul 5 01:05:41 2019 +0800
[FLINK-9311] [pubsub] Improvements to builders + minor improvement to PubSubSink flush logic
---
.../org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
index 7b66577..a960176 100644
--- a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubExample.java
@@ -63,13 +63,13 @@ public class PubSubExample {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(1000L);
- env.addSource(PubSubSource.newBuilder(Integer.class)
+ env.addSource(PubSubSource.newBuilder()
.withDeserializationSchema(new IntegerSerializer())
.withProjectName(projectName)
.withSubscriptionName(subscriptionName)
.build())
.map(PubSubExample::printAndReturn).disableChaining()
- .addSink(PubSubSink.newBuilder(Integer.class)
+ .addSink(PubSubSink.newBuilder()
.withSerializationSchema(new IntegerSerializer())
.withProjectName(projectName)
.withTopicName(outputTopicName).build());
[flink-connector-gcp-pubsub] 06/19: Update version to 1.12-SNAPSHOT
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 55f84bdeec783fe41e720a159319b731485e843e
Author: Piotr Nowojski <pi...@gmail.com>
AuthorDate: Mon May 18 13:36:46 2020 +0200
Update version to 1.12-SNAPSHOT
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index ccd169c..5659dcb 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -24,7 +24,7 @@ under the License.
<parent>
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.11-SNAPSHOT</version>
+ <version>1.12-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
[flink-connector-gcp-pubsub] 18/19: [FLINK-30872][Connector/Google PubSub] Move GCP example to GCP connector repository
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 7d8090b72ffb75727d877534d670a52e3eaa039b
Author: Martijn Visser <ma...@apache.org>
AuthorDate: Wed Feb 1 19:34:12 2023 +0100
[FLINK-30872][Connector/Google PubSub] Move GCP example to GCP connector repository
---
flink-connector-gcp-pubsub/pom.xml | 17 -----------------
flink-examples-streaming-gcp-pubsub/pom.xml | 16 ++++++++++------
pom.xml | 11 +++++++++++
3 files changed, 21 insertions(+), 23 deletions(-)
diff --git a/flink-connector-gcp-pubsub/pom.xml b/flink-connector-gcp-pubsub/pom.xml
index 7288619..1bd75b5 100644
--- a/flink-connector-gcp-pubsub/pom.xml
+++ b/flink-connector-gcp-pubsub/pom.xml
@@ -34,23 +34,6 @@ under the License.
<packaging>jar</packaging>
- <properties>
- <google-cloud-libraries-bom.version>8.1.0</google-cloud-libraries-bom.version>
- </properties>
-
- <dependencyManagement>
- <dependencies>
- <dependency>
- <!-- This is the way we get a consistent set of versions of the Google tools -->
- <groupId>com.google.cloud</groupId>
- <artifactId>libraries-bom</artifactId>
- <version>${google-cloud-libraries-bom.version}</version>
- <type>pom</type>
- <scope>import</scope>
- </dependency>
- </dependencies>
- </dependencyManagement>
-
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index 8720ed9..d66e489 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -22,15 +22,19 @@ under the License.
<modelVersion>4.0.0</modelVersion>
<parent>
- <artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.17-SNAPSHOT</version>
+ <artifactId>flink-connector-gcp-pubsub-parent</artifactId>
+ <version>3.0-SNAPSHOT</version>
</parent>
- <artifactId>flink-examples-streaming-gcp-pubsub_${scala.binary.version}</artifactId>
- <name>Flink : Examples : Build Helper : Streaming Google PubSub</name>
+ <artifactId>flink-examples-streaming-gcp-pubsub</artifactId>
+ <name>Flink : Examples : Streaming Google PubSub</name>
<packaging>jar</packaging>
+ <properties>
+ <japicmp.skip>true</japicmp.skip>
+ </properties>
+
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
@@ -39,8 +43,8 @@ under the License.
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-examples-streaming_${scala.binary.version}</artifactId>
- <version>${project.version}</version>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${flink.version}</version>
</dependency>
</dependencies>
diff --git a/pom.xml b/pom.xml
index 5ca7403..5addf92 100644
--- a/pom.xml
+++ b/pom.xml
@@ -45,10 +45,12 @@ under the License.
<modules>
<module>flink-connector-gcp-pubsub</module>
<module>flink-connector-gcp-pubsub-e2e-tests</module>
+ <module>flink-examples-streaming-gcp-pubsub</module>
</modules>
<properties>
<flink.version>1.16.0</flink.version>
+ <google-cloud-libraries-bom.version>8.1.0</google-cloud-libraries-bom.version>
<junit5.version>5.8.1</junit5.version>
<assertj.version>3.23.1</assertj.version>
@@ -321,6 +323,15 @@ under the License.
<version>2.6</version>
</dependency>
+ <dependency>
+ <!-- This is the way we get a consistent set of versions of the Google tools -->
+ <groupId>com.google.cloud</groupId>
+ <artifactId>libraries-bom</artifactId>
+ <version>${google-cloud-libraries-bom.version}</version>
+ <type>pom</type>
+ <scope>import</scope>
+ </dependency>
+
</dependencies>
</dependencyManagement>
[flink-connector-gcp-pubsub] 08/19: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 3cdd0ac185c3674d754ac16739519dbbc8a0eb6f
Author: Niels Basjes <ni...@basjes.nl>
AuthorDate: Wed Jul 8 13:13:11 2020 +0200
[FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
This closes #12846
---
.../apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
index 8f7bfe6..ef9ad83 100644
--- a/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
+++ b/flink-examples-streaming-gcp-pubsub/src/main/java/org/apache/flink/streaming/examples/gcp/pubsub/PubSubPublisher.java
@@ -19,8 +19,8 @@ package org.apache.flink.streaming.examples.gcp.pubsub;
import com.google.cloud.pubsub.v1.Publisher;
import com.google.protobuf.ByteString;
-import com.google.pubsub.v1.ProjectTopicName;
import com.google.pubsub.v1.PubsubMessage;
+import com.google.pubsub.v1.TopicName;
import java.math.BigInteger;
@@ -44,7 +44,7 @@ class PubSubPublisher {
void publish(int amountOfMessages) {
Publisher publisher = null;
try {
- publisher = Publisher.newBuilder(ProjectTopicName.of(projectName, topicName)).build();
+ publisher = Publisher.newBuilder(TopicName.of(projectName, topicName)).build();
for (int i = 0; i < amountOfMessages; i++) {
ByteString messageData = ByteString.copyFrom(BigInteger.valueOf(i).toByteArray());
PubsubMessage message = PubsubMessage.newBuilder().setData(messageData).build();
[flink-connector-gcp-pubsub] 16/19: [FLINK-24721][build] Remove unnecessary relativePath to parent
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 942a61642f3507e14ad162c85050c077d288adb7
Author: Chesnay Schepler <ch...@apache.org>
AuthorDate: Wed Apr 20 13:55:43 2022 +0200
[FLINK-24721][build] Remove unnecessary relativePath to parent
---
flink-examples-streaming-gcp-pubsub/pom.xml | 1 -
1 file changed, 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index ae32196..b22c935 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -25,7 +25,6 @@ under the License.
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
<version>1.16-SNAPSHOT</version>
- <relativePath>..</relativePath>
</parent>
<artifactId>flink-examples-streaming-gcp-pubsub_${scala.binary.version}</artifactId>
[flink-connector-gcp-pubsub] 07/19: [FLINK-18607][build] Give the maven module a human readable name
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 62d42793281a076c167111c11b6230dca88aa615
Author: Niels Basjes <ni...@basjes.nl>
AuthorDate: Wed Jul 22 11:26:03 2020 +0200
[FLINK-18607][build] Give the maven module a human readable name
This closes #12907
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index 5659dcb..6378010 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -29,7 +29,7 @@ under the License.
</parent>
<artifactId>flink-examples-streaming-gcp-pubsub_${scala.binary.version}</artifactId>
- <name>flink-examples-streaming-gcp-pubsub</name>
+ <name>Flink : Examples : Build Helper : Streaming Google PubSub</name>
<packaging>jar</packaging>
<dependencies>
[flink-connector-gcp-pubsub] 12/19: Update version to 1.15-SNAPSHOT
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit 1c9423fd5d995acf6305ae6843e9ad0ab38fa140
Author: Xintong Song <to...@gmail.com>
AuthorDate: Sat Aug 28 08:33:55 2021 +0800
Update version to 1.15-SNAPSHOT
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index d582b63..b5bcccc 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -24,7 +24,7 @@ under the License.
<parent>
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.14-SNAPSHOT</version>
+ <version>1.15-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
[flink-connector-gcp-pubsub] 05/19: Update version to 1.11-SNAPSHOT
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
commit c93e10dc61c340602cfc424c6a1b526bc17e1f07
Author: Gary Yao <ga...@apache.org>
AuthorDate: Mon Dec 9 17:28:45 2019 +0100
Update version to 1.11-SNAPSHOT
---
flink-examples-streaming-gcp-pubsub/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-examples-streaming-gcp-pubsub/pom.xml b/flink-examples-streaming-gcp-pubsub/pom.xml
index 969f153..ccd169c 100644
--- a/flink-examples-streaming-gcp-pubsub/pom.xml
+++ b/flink-examples-streaming-gcp-pubsub/pom.xml
@@ -24,7 +24,7 @@ under the License.
<parent>
<artifactId>flink-examples-build-helper</artifactId>
<groupId>org.apache.flink</groupId>
- <version>1.10-SNAPSHOT</version>
+ <version>1.11-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>