[jira] [Commented] (RYA-313) Rya Mongo Blows up on Large result sets

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RYA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113665#comment-16113665 ] 

ASF GitHub Bot commented on RYA-313:
------------------------------------

GitHub user isper3at opened a pull request:

    https://github.com/apache/incubator-rya/pull/196

    RYA-313

    Aggregation now is performed over batches of 1000.
    
    ## Description
    >What Changed?
    Changed the iterator to use the non-deprecated Mongo api, enabled the aggregation framework to work properly.
    
    This could be a very good reason to address: [Jira mongo api ticket](https://issues.apache.org/jira/browse/RYA-302)
    
    ### Tests
    >Coverage?
    No tests, just ran locally and assured that no exception for document size was thrown.
    
    ### Links
    [Jira](https://issues.apache.org/jira/browse/RYA-313)
    
    ### Checklist
    - [ ] Code Review
    - [ ] Squash Commits
    
    #### People To Reivew
    @amihalik 


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/isper3at/incubator-rya RYA-313

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-rya/pull/196.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #196
    
----
commit 3b279972198d34a2bcda775bc6ec4f9a87ade8ba
Author: isper3at <sm...@gmail.com>
Date:   2017-08-03T23:03:11Z

    RYA-313
    Aggregation now is performed over batches of 1000.

----


> Rya Mongo Blows up on Large result sets
> ---------------------------------------
>
>                 Key: RYA-313
>                 URL: https://issues.apache.org/jira/browse/RYA-313
>             Project: Rya
>          Issue Type: Bug
>          Components: dao
>    Affects Versions: 3.2.10
>         Environment: Mongo DB with Rya 3.2.11-SNAPSHOT with a lot of data in Rya
>            Reporter: Aaron Mihalik
>            Assignee: Andrew Smith
>
> Simple queries that return a lot of results fail because mongo is trying to send all of the results back at once.  For instance, if I have a lot of data and run something like:
> {noformat}
> SELECT * WHERE 
> {
> 	?s a ?t.
> }
> {noformat}
> I will get this exception.
> {noformat}
> Caused by: com.mongodb.MongoCommandException: Command failed with error 16389: 'aggregation result exceeds maximum document size (16MB)' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "aggregation result exceeds maximum document size (16MB)", "code" : 16389 }
> {noformat}
> I think we need to toss in a "AggregationOptions with Batch = 1000", but I couldn't get that to work immediately.  Somebody with more mongo experience needs to look at this.
> [Here is the line of code|https://github.com/apache/incubator-rya/blob/master/dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/iter/RyaStatementBindingSetCursorIterator.java#L114]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)