You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Ismaël Mejía (JIRA)" <ji...@apache.org> on 2019/05/17 12:49:00 UTC
[jira] [Commented] (BEAM-7256) Add support for allowDiskUse
(AggregationOptions) in MongoDbIO
[ https://issues.apache.org/jira/browse/BEAM-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842152#comment-16842152 ]
Ismaël Mejía commented on BEAM-7256:
------------------------------------
The method that allows to pass the options seems to be available only for the other Java API (the one with `DBCollection` and `DBObject`) but the IO uses the `MongoCollection` one. Not sure if there is a way to pass them, otherwise should be a doable fix. Any idea maybe [~sandboxws]?
> Add support for allowDiskUse (AggregationOptions) in MongoDbIO
> ---------------------------------------------------------------
>
> Key: BEAM-7256
> URL: https://issues.apache.org/jira/browse/BEAM-7256
> Project: Beam
> Issue Type: Improvement
> Components: io-java-mongodb
> Affects Versions: 2.12.0
> Reporter: Javier Cornejo
> Priority: Major
> Attachments: Screen Shot 2019-05-09 at 12.30.51.png
>
>
> When a read is executed over a collection that exceed the memory limit of 104857600 an exception occurs. This is declared by mongodb and is possible to control the error passing a AggregationOptions allowDiskUse true so mongo can sort with disk usage.
> This should be happen only when aggregations are added to read but now is happening even without aggregation at all.
> Please let me know how can help with this improvement / bug.
>
> !Screen Shot 2019-05-09 at 12.30.51.png!
> {code:java}
> PCollection<KV<String, Document>> updateColls = p.apply("Reading Ops Collection: " + key, MongoDbIO .read() .withUri(options.getMongoDBUri()) .withDatabase("local") .withCollection("oplog.rs") .withBucketAuto(true) // .withQueryFn( // FindQuery.create().withFilters( // Filters.and( // Filters.gt("ts", ts.format(dtf)), // Filters.eq("ns", options.getMongoDBDBName() + "" + key), // Filters.eq("op", "u") // ) // ) // // AggregationQuery.create().withMongoDbPipeline(updatedDocsOplogAggregation) // ) )
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)