[jira] [Comment Edited] (BEAM-2639) Unbounded Source for MongoDB

["nevi_me (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268140#comment-16268140 ] 

nevi_me edited comment on BEAM-2639 at 11/28/17 5:25 AM:
---------------------------------------------------------

MongoDB 3.6 will include a `$changeStream` operator, enabling unbounded access to collections without tailing the oplog. [http://mongodb.github.io/mongo-java-driver/3.6/driver/tutorials/change-streams/]

I'm busy preparing to upgrade to MongoDB, I tail the Oplog a lot, and want to implement these change streams. I'd like to contribute an unbounded source for MongoDB, I think it'll help me learn more about Beam's internals. It's not a high priority, but since it'd be my first contribution to ASF, I might need some hand-holding when the time comes.

The one thing I need to first investigate is if there are breaking changes to the way users authenticate to databases. Some drivers have been logging deprecation warnings about upcoming 3.6 changes. I'll also look into that.

I'll provide feedback mid-December when I have downtime.


was (Author: nevi_me):
MongoDB 3.6 will include a `$changeStream` operator, enabling unbounded access to collections. [http://mongodb.github.io/mongo-java-driver/3.6/driver/tutorials/change-streams/]

I'm busy preparing to upgrade to MongoDB, I tail the Oplog a lot, and want to implement these change streams. I'd like to contribute an unbounded source for MongoDB, I think it'll help me learn more about Beam's internals. It's not a high priority, but since it'd be my first contribution to ASF, I might need some hand-holding when the time comes.

The one thing I need to first investigate is if there are breaking changes to the way users authenticate to databases. Some drivers have been logging deprecation warnings about upcoming 3.6 changes. I'll also look into that.

I'll provide feedback mid-December when I have downtime.

> Unbounded Source for MongoDB
> ----------------------------
>
>                 Key: BEAM-2639
>                 URL: https://issues.apache.org/jira/browse/BEAM-2639
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>    Affects Versions: 2.0.0
>            Reporter: nevi_me
>            Assignee: Jean-Baptiste Onofré
>            Priority: Minor
>
> The current MongoDB source is bounded, which means that we can't build streaming pipelines directly from MongoDB.
> MongoDB publishes changes in each collection through the oplog. Would it be possible to create a connector that reads the oplog to create an unbounded source?
> As an oplog is only available through replication, this creates that dependency. We would need to also consider whether a polling method (using the ObjectId) could be an appropriate fallback.
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)