You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 01:54:45 UTC

[GitHub] [beam] kennknowles opened a new issue, #19503: Add support for allowDiskUse (AggregationOptions) in MongoDbIO

kennknowles opened a new issue, #19503:
URL: https://github.com/apache/beam/issues/19503

   When a read is executed over a collection that exceed the memory limit of 104857600 an exception occurs. This is declared by mongodb and is possible to control the error passing a AggregationOptions allowDiskUse true so mongo can sort with disk usage. 
   
   This should be happen only when aggregations are added to read but now is happening even without aggregation at all. 
   
   Please let me know how can help with this improvement /  bug.
   
    
   
   !Screen Shot 2019-05-09 at 12.30.51.png!  
   ```
   
   PCollection<KV<String, Document>> updateColls = p.apply("Reading Ops Collection: " + key, MongoDbIO
   .read() .withUri(options.getMongoDBUri()) .withDatabase("local") .withCollection("oplog.rs") .withBucketAuto(true)
   // .withQueryFn( // FindQuery.create().withFilters( // Filters.and( // Filters.gt("ts", ts.format(dtf)),
   // Filters.eq("ns", options.getMongoDBDBName() + "" + key), // Filters.eq("op", "u") // ) // ) // //
   AggregationQuery.create().withMongoDbPipeline(updatedDocsOplogAggregation) // ) )
   
   ```
   
   
   Imported from Jira [BEAM-7256](https://issues.apache.org/jira/browse/BEAM-7256). Original Jira may contain additional context.
   Reported by: jcornejo.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abdulharisdjafar commented on issue #19503: Add support for allowDiskUse (AggregationOptions) in MongoDbIO

Posted by "abdulharisdjafar (via GitHub)" <gi...@apache.org>.
abdulharisdjafar commented on issue #19503:
URL: https://github.com/apache/beam/issues/19503#issuecomment-1507904364

   Hi There , any update for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org