You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@rya.apache.org by ejwhite922 <gi...@git.apache.org> on 2017/10/04 15:27:57 UTC

[GitHub] incubator-rya pull request #236: RYA-393 MongoDB Batch Writer Duplicate Key ...

GitHub user ejwhite922 opened a pull request:

    https://github.com/apache/incubator-rya/pull/236

    RYA-393 MongoDB Batch Writer Duplicate Key Exception

    ## Description
    Fixed MongoDB DAO Batch Writer from dropping statements when batch contains duplicate keys.
    
    ### Tests
    Integration Tests
    
    ### Links
    [Jira](https://issues.apache.org/jira/browse/RYA-393)
    
    ### Checklist
    - [ ] Code Review
    - [ ] Squash Commits
    
    #### People To Review
    @amihalik 
    @meiercaleb 
    @pujav65 
    @isper3at 


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ejwhite922/incubator-rya RYA-393_MongoBatchWriterDuplicateKeyException

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-rya/pull/236.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #236
    
----

----


---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    
    Refer to this link for build results (access rights to CI server needed): 
    https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/530/



---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by amihalik <gi...@git.apache.org>.

Github user amihalik commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    @ejwhite922 There's a couple issues with the batch writer and it doesn't have to do with exceptions being caught.  (1) the QueueFullChecker thread should not be a daemon thread.... "The Java Virtual Machine exits when the only threads running are all daemon threads" means that the JVM will exit while flushing to mongo.  (2) It doesn't seem like the secondary indexer (i.e. geoindexer) is flushed when I close down the sail layer.  Can you double check that?
    
    Also, it seems like the JVM takes a very long time to exit after I call sailrepository.shutDown().  I think has to do with the BatchWriter as well.


---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    
    Refer to this link for build results (access rights to CI server needed): 
    https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/537/



---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    
    Refer to this link for build results (access rights to CI server needed): 
    https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/523/



---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by pujav65 <gi...@git.apache.org>.

Github user pujav65 commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    wrt to the secondary flushes -- I think there is a jira ticket already for secondary indexers not being flushed appropriately.


---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by amihalik <gi...@git.apache.org>.

Github user amihalik commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    I'm going to hold off on merging this for a minute.  This exception talk has me thinking there are other exceptions that we are not processing correctly. brb


---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by amihalik <gi...@git.apache.org>.

Github user amihalik commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    Asfbot build


---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by pujav65 <gi...@git.apache.org>.

Github user pujav65 commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    Looks good.  @isper3at , yes catching the exception prevents the drops because it will keep going rather than crapping out on individual adds -- that was why there was a catch for that exception in the original dao before we introduced the batch writing stuff.


---

[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...

Posted by amihalik <gi...@git.apache.org>.

Github user amihalik commented on the issue:

    https://github.com/apache/incubator-rya/pull/236
  
    The more I look at this, the more I’m confident that this has to more with the mongo libs and not anything that we wrote for Rya. I’ll try to put together an example tomorrow. 


---