[jira] [Commented] (BEAM-4567) Can't use mongo connector with Atlas MongoDB

["David Billings (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683605#comment-16683605 ] 

David Billings commented on BEAM-4567:
--------------------------------------

Hi, 

This will never work on MongoDB Atlas because they are not going to change the permissions to allow the use of splitVector.

In fact they are actively discouraging it here: [https://jira.mongodb.org/browse/SERVER-10117]  They mention $sample as an alternative but looking at the documents https://docs.mongodb.com/manual/reference/operator/aggregation/sample/ I'm not sure how this replaces the SV functionality. It is a paint point as Atlas is really the preferred way for me to use Beam. 

Problem is in MongoDbIO: 301. 

I'm happy to fix and do a PR but I can't find any alternative to SV which provides the same functionality at the permission level available on Atlas. 

> Can't use mongo connector with Atlas MongoDB
> --------------------------------------------
>
>                 Key: BEAM-4567
>                 URL: https://issues.apache.org/jira/browse/BEAM-4567
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-mongodb
>    Affects Versions: 2.4.0
>         Environment: Google Cloud Dataflow
>            Reporter: Lucas de Sio Rosa
>            Assignee: Jean-Baptiste Onofré
>            Priority: Major
>              Labels: mongodb
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> I can't use the MongoDB connector with a managed Atlas instance. The current implementations makes use of splitVector which is a high-privilege function that cannot be assigned to any user in Atlas.
> An open Jira issue for MongoDB suggests using $sample and $bucketAuto to circunvent this necessity.
> Following is the exception thrown (removed some identifiable information):
> Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.mongodb.MongoCommandException: Command failed with error 13: 'not authorized on <collection> to execute command \{ splitVector: "<collection>.<table>", keyPattern: { _id: 1 }, force: false, maxChunkSize: 1 }' on server <server>. The full response is \{ "ok" : 0.0, "errmsg" : "not authorized on <collection> to execute command { splitVector: \"<collection>.<table>\", keyPattern: { _id: 1 }, force: false, maxChunkSize: 1 }", "code" : 13, "codeName" : "Unauthorized" }
>  
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
>  
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
>  
>  at br.dotz.datalake.ingest.mongodb.MongoDBCollectorPipeline.main(MongoDBCollectorPipeline.java:27)
>  
> Caused by: com.mongodb.MongoCommandException: Command failed with error 13: 'not authorized on <collection> to execute command \{ splitVector: "<collection>.<table>", keyPattern: { _id: 1 }, force: false, maxChunkSize: 1 }' on server <server>. The full response is \{ "ok" : 0.0, "errmsg" : "not authorized on <collection> to execute command { splitVector: \"<collection>.<table>\", keyPattern: { _id: 1 }, force: false, maxChunkSize: 1 }", "code" : 13, "codeName" : "Unauthorized" }
>  
>  at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
>  
>  at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114)
>  
>  at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
>  
>  at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
>  
>  at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:173)
>  
>  at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:215)
>  
>  at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:186)
>  
>  at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:178)
>  
>  at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:91)
>  
>  at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
>  
>  at com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
>  
>  at com.mongodb.Mongo.execute(Mongo.java:772)
>  
>  at com.mongodb.Mongo$2.execute(Mongo.java:759)
>  
>  at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:130)
>  
>  at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:124)
>  
>  at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:114)
>  
>  at org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:332)
>  
>  at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:210)
>  
>  at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:87)
>  
>  at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:62)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)