You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2021/03/02 02:44:11 UTC

[GitHub] [couchdb] kocolosk opened a new pull request #3393: Relax isolation level when indexer reads from DB

kocolosk opened a new pull request #3393:
URL: https://github.com/apache/couchdb/pull/3393


   ## Overview
   
   This patch causes the indexing subsystem to use snapshot isolation when reading from the database. This reduces commit conflicts and ensures the index can make progress even in the case of frequently updated docs.
   
   In the pathological case, a document updated in a fast loop can cause the indexer to stall out entirely when using serializable reads. Each successful update of the doc will cause the indexer to fail to commit if it processed that document. The indexer will retry with a new GRV but the same target DbSeq. In the meantime, our frequently updated document will have advanced beyond `DbSeq` and so the indexer will finish its job without indexing the document at all. This process can be repeated ad infinitum and the document will never actually show up in a view response.
   
   Snapshot reads are safe for this use case precisely because we do have the _changes feed, which guarantees that a concurrent doc update will show up again later in the feed.
   
   As an aside, I'm not entirely happy with the API for changing the isolation level on reads. It seems designed to run an entire transaction at a single isolation level, where I think we often want to be more selectively about modifying isolation levels within a transaction. But I didn't bother suggesting any changes on that front in this PR.
   
   ## Testing recommendations
   
   I posted a rudimentary Python script in a [gist](https://gist.github.com/kocolosk/6a4410b3ee8c6c06ed8d86deb6c76f19). It updates a single document in a loop and tries to run that document through an expensive `map` function. Running this script against `main` should show you the view almost never makes progress; every once in a while it might win the race, but most of the time it is returning a response that does not include recent updates to the document.
   
   Running the script against this branch should show the view essentially keeping up, although the included document will still be a few revisions ahead.
   
   ## Related Issues or Pull Requests
   
   See #3391 for work to make the included doc version consistent with the version observed by the view index function.
   
   ## Checklist
   
   - [x] Code is written and works correctly
   - [x] Changes are covered by tests
   - [ ] Any new configurable parameters are documented in `rel/overlay/etc/default.ini`
   - [ ] A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] kocolosk merged pull request #3393: Relax isolation level when indexer reads from DB

Posted by GitBox <gi...@apache.org>.
kocolosk merged pull request #3393:
URL: https://github.com/apache/couchdb/pull/3393


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] kocolosk commented on pull request #3393: Relax isolation level when indexer reads from DB

Posted by GitBox <gi...@apache.org>.
kocolosk commented on pull request #3393:
URL: https://github.com/apache/couchdb/pull/3393#issuecomment-789258207


   `make check` passed locally


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] nickva commented on pull request #3393: Relax isolation level when indexer reads from DB

Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3393:
URL: https://github.com/apache/couchdb/pull/3393#issuecomment-789059732


   I think this may need the new erlfdb bump. I already created a new release so we can refer to it by tag (v1.2.5). 
   
   ```
   -{erlfdb,           "erlfdb",           {tag, "v1.2.3"}},
   +{erlfdb,           "erlfdb",           {tag, "v1.2.5"}},
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org