You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "nevi_me (JIRA)" <ji...@apache.org> on 2017/07/19 09:27:00 UTC

[jira] [Created] (BEAM-2639) Unbounded Source for MongoDB

nevi_me created BEAM-2639:
-----------------------------

             Summary: Unbounded Source for MongoDB
                 Key: BEAM-2639
                 URL: https://issues.apache.org/jira/browse/BEAM-2639
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
    Affects Versions: 2.0.0
            Reporter: nevi_me
            Assignee: Davor Bonaci
            Priority: Minor


The current MongoDB source is bounded, which means that we can't build streaming pipelines directly from MongoDB.
MongoDB publishes changes in each collection through the oplog. Would it be possible to create a connector that reads the oplog to create an unbounded source?

As an oplog is only available through replication, this creates that dependency. We would need to also consider whether a polling method (using the ObjectId) could be an appropriate fallback.

Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)