[jira] [Created] (BEAM-7256) Add support for allowDiskUse (AggregationOptions) in MongoDbIO

["Javier Cornejo (JIRA)" <ji...@apache.org>]

Javier Cornejo created BEAM-7256:
------------------------------------

             Summary: Add support for allowDiskUse (AggregationOptions) in MongoDbIO 
                 Key: BEAM-7256
                 URL: https://issues.apache.org/jira/browse/BEAM-7256
             Project: Beam
          Issue Type: Improvement
          Components: io-java-mongodb
    Affects Versions: 2.12.0
            Reporter: Javier Cornejo


When a read is executed over a collection that exceed the memory limit of 104857600 an exception occurs. This is declared by mongodb and is possible to control the error passing a AggregationOptions allowDiskUse true so mongo can sort with disk usage. 

This should be happen only when aggregations are added to read but now is happening even without aggregation at all. 

Please let me know how can help with this improvement /  bug.

!image-2019-05-09-12-31-58-137.png!

 
{code:java}
PCollection<KV<String, Document>> updateColls = p.apply("Reading Ops Collection: " + key, MongoDbIO .read() .withUri(options.getMongoDBUri()) .withDatabase("local") .withCollection("oplog.rs") .withBucketAuto(true) // .withQueryFn( // FindQuery.create().withFilters( // Filters.and( // Filters.gt("ts", ts.format(dtf)), // Filters.eq("ns", options.getMongoDBDBName() + "" + key), // Filters.eq("op", "u") // ) // ) // // AggregationQuery.create().withMongoDbPipeline(updatedDocsOplogAggregation) // ) )
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)