You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Richard Deurwaarder (JIRA)" <ji...@apache.org> on 2018/05/07 20:07:00 UTC
[jira] [Comment Edited] (FLINK-9311) PubSub connector
[ https://issues.apache.org/jira/browse/FLINK-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466409#comment-16466409 ]
Richard Deurwaarder edited comment on FLINK-9311 at 5/7/18 8:06 PM:
--------------------------------------------------------------------
PubSub works more like RabbitMQ than Kafka using ack/nack. It has [atleast-once-delivery|https://cloud.google.com/pubsub/docs/subscriber], but using a [message id|http://googleapis.github.io/googleapis/java/all/latest/apidocs/com/google/pubsub/v1/PubsubMessageOrBuilder.html#getMessageId--] exactly-once can be achived.
Is there an existing connector using message id's for deduplication?
was (Author: xeli):
PubSub works more like RabbitMQ than Kafka using ack/nack. It has atleast-once-delivery, but using a [message id|http://googleapis.github.io/googleapis/java/all/latest/apidocs/com/google/pubsub/v1/PubsubMessageOrBuilder.html#getMessageId--] exactly-once can be achived.
Is there an existing connector using message id's for deduplication?
> PubSub connector
> ----------------
>
> Key: FLINK-9311
> URL: https://issues.apache.org/jira/browse/FLINK-9311
> Project: Flink
> Issue Type: New Feature
> Components: Streaming Connectors
> Reporter: Richard Deurwaarder
> Priority: Minor
>
> I would like start adding some google cloud connectors starting with a PubSub Source. I have a basic implementation ready but I want it to be able to:
> * easily scale up (should I have it extend RichParallelSourceFunction for this?)
> * Make it easier to provide the google cloud credentials. This would require being able to send some json string / ServiceAccount to the nodes when starting up this source.
> Could this be something that would be useful for others and added to the flink connectors repo?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)