[GitHub] [couchdb-nano] eridal opened a new issue #215: Follow: analysis of events

[GitBox <gi...@apache.org>]

eridal opened a new issue #215: Follow: analysis of events
URL: https://github.com/apache/couchdb-nano/issues/215
 
 
   In the search of some bug in my code I've made an analysis about the [follow module](https://github.com/cloudant-labs/cloudant-follow).
   
   I've opened https://github.com/cloudant-labs/cloudant-follow/issues/141 with my findings, and thought on sharing here since the library is a directly nano's dependency.
   
   Please feel free to comment either on this issue or in the follow's one.
   
   ## Expected Behavior
   
   The current state of follow's events could do some beef up.
   
   There are undocumented events, or events that can surprisingly happen multiple times (ie: multiple `start`)
   
   On top of that the way the library handles the timeouts is not rock solid. For instance, my current problem was due to a combination of the way that `heartbeat` and `innactivity_ms` options are used; those have different usages and internal rules which are not self-evident from the outside, and the outcome is unexpected behavior, like `restart` event firing instead of `timeout`.
   
   ## Current Behavior
   
   The in-depth analysis was added to the follow's repo. I'm creating this issue for visibility-wise.
   
   link: https://github.com/cloudant-labs/cloudant-follow/issues/141
   
   ## Possible Solution
   
   One option is to beef up the current follow package; that is to decide a set of events with better semantics and unambiguous conditions, implement them into the current code, and then release a major version.
   
   Another option is to replace that library. Using that same proposed set of events and semantics, either start from scratch or fork the library. This could also be a good opportunity for apache to adopt such fork.
   
   Anyways I think I'm not the right person to decide about that, and please take my suggestions just like ideas.
   
   ## Steps to Reproduce (for bugs)
   
   ````js
   const nano = require('nano')
   
   !(async () => {
   
     const db = nano('https://replicate.npmjs.com')
     const feed = await db.use('').follow({
       since: 3261400, // just a bit before a problematic change
       include_docs: true,
       hearteat: 60 * 1000,
       inactivity_ms: 5 * 1000,
     })
   
     let changes = 0
   
     feed.on('change', (change) => {
       console.log('change', {
         seq: change.seq,
         count: changes += 1
       })
     })
   
     // from https://github.com/cloudant-labs/cloudant-follow#events
     const events = [
       'start',
       'confirm_request',
       'confirm',
       'catchup',
       'wait',
       'timeout',
       'retry',
       'stop',
       'error',
     ]
   
     for (const event of events) {
       feed.on(event, () => {
         console.log(event)
       })
     }
   
     feed.follow()
   
   })()
   ````
   
   ---
   
   ## Context
   
   My issue appeared trying to consume the `/_changes/` from the npm's registry database. There is a current process which runs daily and downloads 5M+ sequences, which I'm currently replacing with an incremental build approach.
   
   This new incremental build will be used by two different scenarios:
   
   1. **with an empty dataset**: the process starts consuming changes from genesis, with an empty dataset, and it will stop once it is catchup. While this will be infrequently executed, it will be useful for re-processing the entire database in case something goes wrong.
   
   2. **the incremental build**: the process starts consuming since the last update, and requires only the delta between the last run and the current database state. This will be the most common use case and possibly will be run multiple times per day.
   
   
   ## Your Environment
   * Version used: v8.2.2
   * Browser Name and version: nodejs v10.15.2
   * Operating System and version (desktop or mobile): linux
   * Link to your project: https://github.com/nice-registry/all-the-package-repos
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [couchdb-nano] ricellis commented on issue #215: Follow: analysis of events

[GitBox <gi...@apache.org>]

ricellis commented on issue #215: Follow: analysis of events
URL: https://github.com/apache/couchdb-nano/issues/215#issuecomment-600537075
 
 
   See #208 - because of the deprecation of the `request` module the plan for Nano is to no longer use the `cloudant-follow` dependency for changes reading functionality.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [couchdb-nano] eridal closed issue #215: Follow: analysis of events

[GitBox <gi...@apache.org>]

eridal closed issue #215: Follow: analysis of events
URL: https://github.com/apache/couchdb-nano/issues/215
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services