You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/07/07 15:55:18 UTC
[GitHub] [flink] nielsbasjes opened a new pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
nielsbasjes opened a new pull request #12846:
URL: https://github.com/apache/flink/pull/12846
## What is the purpose of the change
The PubSub connector was created with the Google Cloud libraries of some time ago.
Because over time Google has changed their API in a not backward compatible way this change simply updates the Google PubSub dependencies and fixes the API incompatibilities.
## Brief change log
- Use the updated dependencies from Google
- Fix problematic API changes
- .usePlaintext(true)
- NoCredentials.getInstance()
- Limit the retries in case of testing.
- Apparently now in the Google code a retry system is present that does way too many retries for testing scenarios.
## Verifying this change
This change is already covered by existing tests, such as the tests in the **flink-connector-gcp-pubsub** and **flink-connector-gcp-pubsub-emulator-tests** maven modules.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **YES**
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-658270987
@Xeli @becketqin @rmetzger Since you were involved in reviewing this component when originally created I'm tagging you to kindly review these changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-662299499
Once the documentation has been updated, I'm +1 on merging this PR!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b3a4da61a337a80a70a5125ed135d3095def41e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337)
* 5e5efbc4355a98d5240f69a03bd014136c019c2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454)
* f903266eaf975ba1f31fb4043a7616309193b99b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654960417
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 51e61b77cf26f37a3832dc8d0dde720417b15bc5 (Tue Jul 07 15:57:42 UTC 2020)
**Warnings:**
* **3 pom.xml files were touched**: Check for build and licensing issues.
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18448).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on a change in pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on a change in pull request #12846:
URL: https://github.com/apache/flink/pull/12846#discussion_r457948500
##########
File path: flink-connectors/flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSink.java
##########
@@ -97,17 +100,31 @@ public void open(Configuration configuration) throws Exception {
serializationSchema.open(() -> getRuntimeContext().getMetricGroup().addGroup("user"));
Publisher.Builder builder = Publisher
- .newBuilder(ProjectTopicName.of(projectName, topicName))
+ .newBuilder(TopicName.of(projectName, topicName))
.setCredentialsProvider(FixedCredentialsProvider.create(credentials));
+ // Having the host and port for the emulator means we are in a testing scenario.
if (hostAndPortForEmulator != null) {
managedChannel = ManagedChannelBuilder
.forTarget(hostAndPortForEmulator)
- .usePlaintext(true) // This is 'Ok' because this is ONLY used for testing.
+ .usePlaintext() // This is 'Ok' because this is ONLY used for testing.
.build();
channel = GrpcTransportChannel.newBuilder().setManagedChannel(managedChannel).build();
builder.setChannelProvider(FixedTransportChannelProvider.create(channel))
- .setCredentialsProvider(NoCredentialsProvider.create());
+ .setCredentialsProvider(EmulatorCredentialsProvider.create())
+ // In test scenarios we are limiting the Retry settings.
+ // The values here are based on the default settings with lower attempts and timeouts.
+ .setRetrySettings(
+ RetrySettings.newBuilder()
+ .setMaxAttempts(10)
+ .setTotalTimeout(Duration.ofSeconds(10))
+ .setInitialRetryDelay(Duration.ofMillis(100))
+ .setRetryDelayMultiplier(1.3)
+ .setMaxRetryDelay(Duration.ofSeconds(5))
+ .setInitialRpcTimeout(Duration.ofSeconds(5))
+ .setRpcTimeoutMultiplier(1)
+ .setMaxRpcTimeout(Duration.ofSeconds(10))
+ .build());
Review comment:
I chose to deviate from the existing code as little as possible.
My primary focus is to update to the latest Google dependencies and to make sure it all really works.
So these changes in the retry settings were put in a codepath that is only touched when using the emulator and thus testing scenario.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e5efbc4355a98d5240f69a03bd014136c019c2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454)
* f903266eaf975ba1f31fb4043a7616309193b99b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbd3e1559f4a80424cbade77ed3034977943bf0b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463)
* 8c8f594927a2b72c23f8619c1932219585755054 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f903266eaf975ba1f31fb4043a7616309193b99b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457)
* fbd3e1559f4a80424cbade77ed3034977943bf0b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51e61b77cf26f37a3832dc8d0dde720417b15bc5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308)
* 1bc023eddd47a20982023029ec32b54704c1744f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #12846:
URL: https://github.com/apache/flink/pull/12846#discussion_r457381278
##########
File path: flink-connectors/flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSink.java
##########
@@ -273,7 +290,13 @@ private PubSubSinkBuilder(SerializationSchema<IN> serializationSchema) {
*/
public PubSubSink<IN> build() throws IOException {
if (credentials == null) {
- credentials = defaultCredentialsProviderBuilder().build().getCredentials();
+ if (hostAndPort == null) {
+ // No hostAndPort is the normal scenario so we use the default credentials.
+ credentials = defaultCredentialsProviderBuilder().build().getCredentials();
+ } else {
+ // With hostAndPort the PubSub emulator is used so we do not have credentials.
+ credentials = EmulatorCredentials.getInstance();
+ }
Review comment:
Again, I guess credentials handling should be done outside of the connector (so that we can pass different credentials for the test)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f903266eaf975ba1f31fb4043a7616309193b99b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457)
* fbd3e1559f4a80424cbade77ed3034977943bf0b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51e61b77cf26f37a3832dc8d0dde720417b15bc5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f903266eaf975ba1f31fb4043a7616309193b99b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b3a4da61a337a80a70a5125ed135d3095def41e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337)
* 5e5efbc4355a98d5240f69a03bd014136c019c2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454)
* f903266eaf975ba1f31fb4043a7616309193b99b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-661054981
Answering my own question: It seems that I need to add
```
<dependencyManagement>
<dependencies>
<dependency>
<!-- This is the way we get a consistent set of versions of the Google tools -->
<groupId>com.google.cloud</groupId>
<artifactId>libraries-bom</artifactId>
<version>8.0.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
```
to my own project now.
Without it, I get the following error message:
```
Caused by: java.lang.AbstractMethodError: Receiver class com.google.api.gax.grpc.InstantiatingGrpcChannelProvider does not define or inherit an implementation of the resolved method 'abstract com.google.api.gax.rpc.TransportChannelProvider withExecutor(java.util.concurrent.Executor)' of interface com.google.api.gax.rpc.TransportChannelProvider.
at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:140)
at com.google.cloud.pubsub.v1.stub.GrpcPublisherStub.create(GrpcPublisherStub.java:195)
at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:187)
at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:88)
at com.google.cloud.pubsub.v1.Publisher$Builder.build(Publisher.java:798)
at org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSink.open(PubSubSink.java:130)
```
Can't we somehow define the libraries-bom as the "default" bom?
If we can't, then we should update the pubsub connector's documentation page
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51e61b77cf26f37a3832dc8d0dde720417b15bc5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbd3e1559f4a80424cbade77ed3034977943bf0b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463)
* 8c8f594927a2b72c23f8619c1932219585755054 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #12846:
URL: https://github.com/apache/flink/pull/12846#discussion_r458597136
##########
File path: flink-connectors/flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSink.java
##########
@@ -97,17 +100,31 @@ public void open(Configuration configuration) throws Exception {
serializationSchema.open(() -> getRuntimeContext().getMetricGroup().addGroup("user"));
Publisher.Builder builder = Publisher
- .newBuilder(ProjectTopicName.of(projectName, topicName))
+ .newBuilder(TopicName.of(projectName, topicName))
.setCredentialsProvider(FixedCredentialsProvider.create(credentials));
+ // Having the host and port for the emulator means we are in a testing scenario.
if (hostAndPortForEmulator != null) {
managedChannel = ManagedChannelBuilder
.forTarget(hostAndPortForEmulator)
- .usePlaintext(true) // This is 'Ok' because this is ONLY used for testing.
+ .usePlaintext() // This is 'Ok' because this is ONLY used for testing.
.build();
channel = GrpcTransportChannel.newBuilder().setManagedChannel(managedChannel).build();
builder.setChannelProvider(FixedTransportChannelProvider.create(channel))
- .setCredentialsProvider(NoCredentialsProvider.create());
+ .setCredentialsProvider(EmulatorCredentialsProvider.create())
+ // In test scenarios we are limiting the Retry settings.
+ // The values here are based on the default settings with lower attempts and timeouts.
+ .setRetrySettings(
+ RetrySettings.newBuilder()
+ .setMaxAttempts(10)
+ .setTotalTimeout(Duration.ofSeconds(10))
+ .setInitialRetryDelay(Duration.ofMillis(100))
+ .setRetryDelayMultiplier(1.3)
+ .setMaxRetryDelay(Duration.ofSeconds(5))
+ .setInitialRpcTimeout(Duration.ofSeconds(5))
+ .setRpcTimeoutMultiplier(1)
+ .setMaxRpcTimeout(Duration.ofSeconds(10))
+ .build());
Review comment:
You are right, it would be an unrelated refactoring to address this issue (I personally try to reduce code smells like this when I have the opportunity to do it, but I understand that keeping the scope of a change small and self-contained is also valuable).
I've filed a JIRA ticket to address this in the future: https://issues.apache.org/jira/browse/FLINK-18669
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c8f594927a2b72c23f8619c1932219585755054 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
}, {
"hash" : "3acc275762c0fef48751d689efe249423be96583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526",
"triggerID" : "3acc275762c0fef48751d689efe249423be96583",
"triggerType" : "PUSH"
}, {
"hash" : "bf1d42483f7aba45d30878736e8503a3d3ddf91f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4695",
"triggerID" : "bf1d42483f7aba45d30878736e8503a3d3ddf91f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf1d42483f7aba45d30878736e8503a3d3ddf91f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4695)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-661052681
I'm not an expert of Maven's BOM mechanism, hence the following question:
> At this point I chose to make it such that downstream projects must specify which libraries-bom version they need. Although this is an extra step for users I hope this will make updating to newer versions of the Google tools easier.
isn't the gcp connector defining libraries-bom 8.0.0 as the default version the connector expects? If users want to change it, they need to define a different bom version?
Or do users always need to define the libraries-bom themselves as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-660987429
I pushed a rebased version of this PR to my personal Azure to make sure the E2E tests are passing after this change: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8218&view=results
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-658267581
Notes about the current set of changes:
- Updated the Google dependencies. All Google dependencies no longer specify any versions. So in all cases the inclusion of the `libraries-bom` is needed. At this point I chose to make it such that downstream projects must specify which `libraries-bom` version they need. Although this is an extra step for users I hope this will make updating to newer versions of the Google tools easier.
- I fixed almost all cases of Google/Flink api's that have become deprecated over time. One clear exception: The use of the ListCheckpointed. This seemed to become too large ( a full rewrite of the handling of the acknowledgements ) so I think it should be a separate effort.
- I replaced the use of the NoCredentials and NoCredentialsProvider with custom implementations because I ran into problems. See also my bug report towards Google https://github.com/googleapis/gax-java/issues/1148
- I added a full end-to-end test that both reads from and writes to PubSub.
- I updated the tests to run in parallel 4 (instead of 1) which surfaced a small bug that only appears on "ending" streams (like we only have in test scenarios). Solution: Move the closing of the connection to a slightly later point. This also meant that the `BoundedStringDeserializer` which counted the number of messages could no longer be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
}, {
"hash" : "3acc275762c0fef48751d689efe249423be96583",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526",
"triggerID" : "3acc275762c0fef48751d689efe249423be96583",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3acc275762c0fef48751d689efe249423be96583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-662013420
I did a lot of digging today and in the end it turns out that there was a dependency problem in the libraries-bom 8.0.0 that has been fixed in the 8.1.0 version.
So simply upgrading and removing all the needless other things was the solution that works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger closed pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger closed pull request #12846:
URL: https://github.com/apache/flink/pull/12846
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
}, {
"hash" : "3acc275762c0fef48751d689efe249423be96583",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526",
"triggerID" : "3acc275762c0fef48751d689efe249423be96583",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c8f594927a2b72c23f8619c1932219585755054 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509)
* 3acc275762c0fef48751d689efe249423be96583 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1bc023eddd47a20982023029ec32b54704c1744f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331)
* 5b3a4da61a337a80a70a5125ed135d3095def41e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51e61b77cf26f37a3832dc8d0dde720417b15bc5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-662947822
Yes, sorry! I was somehow assuming that the libraries-bom is still required (I thought you only resolved the AbstractMethodError I reported).
I will merge this PR before the end of this week!
Thanks a lot for your contribution!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #12846:
URL: https://github.com/apache/flink/pull/12846#discussion_r457380452
##########
File path: flink-connectors/flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSink.java
##########
@@ -97,17 +100,31 @@ public void open(Configuration configuration) throws Exception {
serializationSchema.open(() -> getRuntimeContext().getMetricGroup().addGroup("user"));
Publisher.Builder builder = Publisher
- .newBuilder(ProjectTopicName.of(projectName, topicName))
+ .newBuilder(TopicName.of(projectName, topicName))
.setCredentialsProvider(FixedCredentialsProvider.create(credentials));
+ // Having the host and port for the emulator means we are in a testing scenario.
if (hostAndPortForEmulator != null) {
managedChannel = ManagedChannelBuilder
.forTarget(hostAndPortForEmulator)
- .usePlaintext(true) // This is 'Ok' because this is ONLY used for testing.
+ .usePlaintext() // This is 'Ok' because this is ONLY used for testing.
.build();
channel = GrpcTransportChannel.newBuilder().setManagedChannel(managedChannel).build();
builder.setChannelProvider(FixedTransportChannelProvider.create(channel))
- .setCredentialsProvider(NoCredentialsProvider.create());
+ .setCredentialsProvider(EmulatorCredentialsProvider.create())
+ // In test scenarios we are limiting the Retry settings.
+ // The values here are based on the default settings with lower attempts and timeouts.
+ .setRetrySettings(
+ RetrySettings.newBuilder()
+ .setMaxAttempts(10)
+ .setTotalTimeout(Duration.ofSeconds(10))
+ .setInitialRetryDelay(Duration.ofMillis(100))
+ .setRetryDelayMultiplier(1.3)
+ .setMaxRetryDelay(Duration.ofSeconds(5))
+ .setInitialRpcTimeout(Duration.ofSeconds(5))
+ .setRpcTimeoutMultiplier(1)
+ .setMaxRpcTimeout(Duration.ofSeconds(10))
+ .build());
Review comment:
It seems a bit problematic to have testing code in the main sink code: Wouldn't it be generally a good idea to allow the connector's users to configure the retry (and possibly all settings) themselves?
In my experience with the connectors, as more and more people start using them, they'll demand for access to all configuration options.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51e61b77cf26f37a3832dc8d0dde720417b15bc5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308)
* 1bc023eddd47a20982023029ec32b54704c1744f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
}, {
"hash" : "3acc275762c0fef48751d689efe249423be96583",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3acc275762c0fef48751d689efe249423be96583",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c8f594927a2b72c23f8619c1932219585755054 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509)
* 3acc275762c0fef48751d689efe249423be96583 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1bc023eddd47a20982023029ec32b54704c1744f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331)
* 5b3a4da61a337a80a70a5125ed135d3095def41e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-661778583
I have been struggling with this BOM mechanism.
Apparently this mechanism allows importing the dependency management parts from other dependencies (that must be `<packaging>pom</packaging>`).
- https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Importing_Dependencies
- https://stackoverflow.com/a/11779302/114196
- https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
The way Google uses this system is that the dependencies you include from Google have no versions anymore at all.
The actual versions that are to be used are pulled from the specified libraries-bom.
Note that this libraries-bom imports the bom from various other parts like the google-cloud-bom.
And the google-cloud-bom imports the google-cloud-core-bom.
etc. ...
I tried several things. When I simply add the libraries-bom as a dependency then it will not find the versions specified in the 'sub-included' boms (like the google-cloud-bom I mentioned).
When my project does not have the libraries-bom then it fails not knowing the desired dependency versions.
I have not been able to improve this setup.
So I agree this should be part of the documentation.
P.S. What should I do regarding the Chinese version of this? Should I simply insert the English variant and let someone else translate it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
}, {
"hash" : "3acc275762c0fef48751d689efe249423be96583",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526",
"triggerID" : "3acc275762c0fef48751d689efe249423be96583",
"triggerType" : "PUSH"
}, {
"hash" : "bf1d42483f7aba45d30878736e8503a3d3ddf91f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf1d42483f7aba45d30878736e8503a3d3ddf91f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3acc275762c0fef48751d689efe249423be96583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526)
* bf1d42483f7aba45d30878736e8503a3d3ddf91f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-662299132
Thanks a lot for revisiting the dependency problem!
> What should I do regarding the Chinese version of this? Should I simply insert the English variant and let someone else translate it?
Since the Chinese version has been translated already, you'll need to file a JIRA ticket to update the translation (ideally include a link to your commit to make it easier.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on a change in pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on a change in pull request #12846:
URL: https://github.com/apache/flink/pull/12846#discussion_r457943087
##########
File path: flink-connectors/flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSink.java
##########
@@ -273,7 +290,13 @@ private PubSubSinkBuilder(SerializationSchema<IN> serializationSchema) {
*/
public PubSubSink<IN> build() throws IOException {
if (credentials == null) {
- credentials = defaultCredentialsProviderBuilder().build().getCredentials();
+ if (hostAndPort == null) {
+ // No hostAndPort is the normal scenario so we use the default credentials.
+ credentials = defaultCredentialsProviderBuilder().build().getCredentials();
+ } else {
+ // With hostAndPort the PubSub emulator is used so we do not have credentials.
+ credentials = EmulatorCredentials.getInstance();
+ }
Review comment:
You can already pass different credentials if you want to via the withCredentials method in both the normal and the emulator situations.
This code is about which are the right **default** credentials to choose in case none were provided: Either the default Google credentials or (in case the emulator was specified) the "Emulator Credentials".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on a change in pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on a change in pull request #12846:
URL: https://github.com/apache/flink/pull/12846#discussion_r457937728
##########
File path: flink-connectors/flink-connector-gcp-pubsub/pom.xml
##########
@@ -56,18 +56,22 @@ under the License.
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.google.cloud</groupId>
+ <artifactId>google-cloud-core</artifactId>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
+ </dependency>
+
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsub</artifactId>
- <!-- Version is pulled from google-cloud-bom -->
- <exclusions>
- <!-- Exclude an old version of guava that is being pulled
- in by a transitive dependency of google-api-client -->
- <exclusion>
- <groupId>com.google.guava</groupId>
- <artifactId>guava-jdk5</artifactId>
- </exclusion>
- </exclusions>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.api.grpc</groupId>
+ <artifactId>grpc-google-cloud-pubsub-v1</artifactId>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
Review comment:
No, this one is really needed. I also looked into this while writing it.
Apparently this connector uses in a few places the com.google.pubsub.v1.SubscriberGrpc which is not included in the dependencies otherwise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #12846:
URL: https://github.com/apache/flink/pull/12846#discussion_r458593290
##########
File path: flink-connectors/flink-connector-gcp-pubsub/pom.xml
##########
@@ -56,18 +56,22 @@ under the License.
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.google.cloud</groupId>
+ <artifactId>google-cloud-core</artifactId>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
+ </dependency>
+
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsub</artifactId>
- <!-- Version is pulled from google-cloud-bom -->
- <exclusions>
- <!-- Exclude an old version of guava that is being pulled
- in by a transitive dependency of google-api-client -->
- <exclusion>
- <groupId>com.google.guava</groupId>
- <artifactId>guava-jdk5</artifactId>
- </exclusion>
- </exclusions>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.api.grpc</groupId>
+ <artifactId>grpc-google-cloud-pubsub-v1</artifactId>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
Review comment:
Mh, ok :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-660980257
Thank you for opening this PR. I'll review it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
}, {
"hash" : "8c8f594927a2b72c23f8619c1932219585755054",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509",
"triggerID" : "8c8f594927a2b72c23f8619c1932219585755054",
"triggerType" : "PUSH"
}, {
"hash" : "3acc275762c0fef48751d689efe249423be96583",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526",
"triggerID" : "3acc275762c0fef48751d689efe249423be96583",
"triggerType" : "PUSH"
}, {
"hash" : "bf1d42483f7aba45d30878736e8503a3d3ddf91f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4695",
"triggerID" : "bf1d42483f7aba45d30878736e8503a3d3ddf91f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3acc275762c0fef48751d689efe249423be96583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4526)
* bf1d42483f7aba45d30878736e8503a3d3ddf91f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4695)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-658267581
Notes about the current set of changes:
- Updated the Google dependencies. All Google dependencies no longer specify any versions. So in all cases the inclusion of the `libraries-bom` is needed. At this point I chose to make it such that downstream projects must specify which `libraries-bom` version they need. Although this is an extra step for users I hope this will make updating to newer versions of the Google tools easier.
- I fixed almost all cases of Google/Flink api's that have become deprecated over time. One clear exception: The use of the ListCheckpointed. This seemed to become too large ( a full rewrite of the handling of the acknowledgements ) so I think it should be a separate effort.
- I replaced the use of the NoCredentials and NoCredentialsProvider with custom implementations because I ran into problems. See also my bug report towards Google https://github.com/googleapis/gax-java/issues/1148
- I added a full end-to-end test that both reads from and writes to PubSub.
- I updated the tests to run in parallel 4 (instead of 1) which surfaced a small bug that only appears on "ending" streams (like we only have in test scenarios). Solution: Move the closing of the connection to a slightly later point.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
}, {
"hash" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4457",
"triggerID" : "f903266eaf975ba1f31fb4043a7616309193b99b",
"triggerType" : "PUSH"
}, {
"hash" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463",
"triggerID" : "fbd3e1559f4a80424cbade77ed3034977943bf0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbd3e1559f4a80424cbade77ed3034977943bf0b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b3a4da61a337a80a70a5125ed135d3095def41e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337)
* 5e5efbc4355a98d5240f69a03bd014136c019c2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4454)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
}, {
"hash" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e5efbc4355a98d5240f69a03bd014136c019c2b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b3a4da61a337a80a70a5125ed135d3095def41e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337)
* 5e5efbc4355a98d5240f69a03bd014136c019c2b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #12846:
URL: https://github.com/apache/flink/pull/12846#discussion_r457330265
##########
File path: flink-connectors/flink-connector-gcp-pubsub/pom.xml
##########
@@ -56,18 +56,22 @@ under the License.
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.google.cloud</groupId>
+ <artifactId>google-cloud-core</artifactId>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
+ </dependency>
+
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsub</artifactId>
- <!-- Version is pulled from google-cloud-bom -->
- <exclusions>
- <!-- Exclude an old version of guava that is being pulled
- in by a transitive dependency of google-api-client -->
- <exclusion>
- <groupId>com.google.guava</groupId>
- <artifactId>guava-jdk5</artifactId>
- </exclusion>
- </exclusions>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.api.grpc</groupId>
+ <artifactId>grpc-google-cloud-pubsub-v1</artifactId>
+ <!-- Version is pulled from google-cloud-bom (loaded via the libraries-bom) -->
Review comment:
Why is this dependency needed? Isn't `google-cloud-pubsub` enough?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4308",
"triggerID" : "51e61b77cf26f37a3832dc8d0dde720417b15bc5",
"triggerType" : "PUSH"
}, {
"hash" : "1bc023eddd47a20982023029ec32b54704c1744f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4331",
"triggerID" : "1bc023eddd47a20982023029ec32b54704c1744f",
"triggerType" : "PUSH"
}, {
"hash" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337",
"triggerID" : "5b3a4da61a337a80a70a5125ed135d3095def41e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b3a4da61a337a80a70a5125ed135d3095def41e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4337)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-662940858
As it is now (i.e. the libraries-bom issue resolved) I do not think any documentation changes are needed.
@rmetzger Do you agree?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] nielsbasjes commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #12846:
URL: https://github.com/apache/flink/pull/12846#issuecomment-661808942
I did a bit more testing.
Turns out the directly included dependencies get an explicit version in the final pom.
Even though the libraries-bom is present it seems unused when resolving the versions in my downstream project.
So on my machine the `~/.m2/repository/org/apache/flink/flink-connector-gcp-pubsub_2.11/1.12-SNAPSHOT/flink-connector-gcp-pubsub_2.11-1.12-SNAPSHOT.pom` has this:
```
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-core</artifactId>
<version>1.93.6</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsub</artifactId>
<version>1.107.0</version>
<scope>compile</scope>
</dependency>
```
and also
```
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>libraries-bom</artifactId>
<version>8.0.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org