[I] couch changes deadline [couchdb]

["ronag (via GitHub)" <gi...@apache.org>]

ronag opened a new issue, #5224:
URL: https://github.com/apache/couchdb/issues/5224

   Currently when using changes with a filter it can take a very long time before a change is received which makes it difficult to create checkpoints to start from in case the change feed is prematurely restarted.
   
   e.g. if you have a huge database with and a very specific change feed selector it can take 8 hours before the first change arrives. If the service has a tendency (for whatever reason) to restart once an hour it will never be able to make progress since no checkpoint ever occurs.
   
   It would be nice if "normal" change feeds would have an option that says "if x duration has passed" then end the change feed and return the last progress equence number.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] couch changes deadline [couchdb]

["nickva (via GitHub)" <gi...@apache.org>]

nickva commented on issue #5224:
URL: https://github.com/apache/couchdb/issues/5224#issuecomment-2334273684

   Yeah agree. This is a long-standing issue. Another idea might be emit periodic markers (a row without a doc_id, or empty doc_id or something) with the current update sequence, so a checkpointer can checkpoint intermediate results. We already emit a `no_pass` ping from the node workers to the coordinator to keep the stream alive, we just don't emit it to the users.
   
   A timeout might work as well, but there is a danger there if we don't differentiate between a timeout expiring and finishing streaming.  It could lead to a sort of a data loss then, as users would consider all the data streamed when in fact we just timed-out early.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org