You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Nick Vatamaniuc <va...@apache.org> on 2020/02/19 22:39:45 UTC
[DISCUSS] Streaming API in CouchDB 4.0
Hello everyone,
I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
By "streaming APIs" I mean APIs which stream data in row as it gets
read from the database. These are the endpoints I was thinking of:
_all_docs, _all_dbs, _dbs_info and query results
I want to focus on what happens when FoundationDB transactions
time-out after 5 seconds. Currently, all those APIs except _changes[1]
feeds, will crash or freeze. The reason is because the
transaction_too_old error at the end of 5 seconds is retry-able by
default, so the request handlers run again and end up shoving the
whole request down the socket again, headers and all, which is
obviously broken and not what we want.
There are few alternatives discussed in couchdb-dev channel. I'll
present some behaviors but feel free to add more. Some ideas might
have been discounted on the IRC discussion already but I'll present
them anyway in case is sparks further conversation:
A) Do what _changes[1] feeds do. Start a new transaction and continue
streaming the data from the next key after last emitted in the
previous transaction. Document the API behavior change that it may
present a view of the data is never a point-in-time[4] snapshot of the
DB.
- Keeps the API shape the same as CouchDB <4.0. Client libraries
don't have to change to continue using these CouchDB 4.0 endpoints
- This is the easiest to implement since it would re-use the
implementation for _changes feed (an extra option passed to the fold
function).
- Breaks API behavior if users relied on having a point-in-time[4]
snapshot view of the data.
B) Simply end the stream. Let the users pass a `?transaction=true`
param which indicates they are aware the stream may end early and so
would have to paginate from the last emitted key with a skip=1. This
will keep the request bodies the same as current CouchDB. However, if
the users got all the data one request, they will end up wasting
another request to see if there is more data available. If they didn't
get any data they might have a too large of a skip value (see [2]) so
would have to guess different values for start/end keys. Or impose max
limit for the `skip` parameter.
C) End the stream and add a final metadata row like a "transaction":
"timeout" at the end. That will let the user know to keep paginating
from the last key onward. This won't work for `_all_dbs` and
`_dbs_info`[3] Maybe let those two endpoints behave like _changes
feeds and only use this for views and and _all_docs? If we like this
choice, let's think what happens for those as I couldn't come up with
anything decent there.
D) Same as C but to solve the issue with skips[2], emit a bookmark
"key" of where the iteration stopped and the current "skip" and
"limit" params, which would keep decreasing. Then user would pass
those in "start_key=..." in the next request along with the limit and
skip params. So something like "continuation":{"skip":599, "limit":5,
"key":"..."}. This has the same issue with array results for
`_all_dbs` and `_dbs_info`[3].
E) Enforce low `limit` and `skip` parameters. Enforce maximum values
there such that response time is likely to fit in one transaction.
This could be tricky as different runtime environments will have
different characteristics. Also, if the timeout happens there isn't a
a nice way to send an HTTP error since we already sent the 200
response. The downside is that this might break how some users use the
API, if say the are using large skips and limits already. Perhaps here
we do both B and D, such that if users want transactional behavior,
they specify that `transaction=true` param and only then we enforce
low limit and skip maximums.
F) At least for `_all_docs` it seems providing a point-in-time
snapshot view doesn't necessarily need to be tied to transaction
boundaries. We could check the update sequence of the database at the
start of the next transaction and if it hasn't changed we can continue
emitting a consistent view. This can apply to C and D and would just
determine when the stream ends. If there are no writes happening to
the db, this could potential streams all the data just like option A
would do. Not entirely sure if this would work for views.
So what do we think? I can see different combinations of options here,
maybe even different for each API point. For example `_all_dbs`,
`_dbs_info` are always A, and `_all_docs` and views default to A but
have parameters to do F, etc.
Cheers,
-Nick
Some footnotes:
[1] _changes feeds is the only one that works currently. It behaves as
per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
That is, we continue streaming the data by resetting the transaction
object and restarting from the last emitted key (db sequence in this
case). However, because the transaction restarts if a document is
updated while the streaming take place, it may appear in the _changes
feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
have to document it, since previously we presented this point-in-time
snapshot of the database from when we started streaming.
[2] Our streaming APIs have both skips and limits. Since FDB doesn't
currently support efficient offsets for key selectors
(https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
we implemented skip by iterating over the data. This means that a skip
of say 100000 could keep timing out the transaction without yielding
any data.
[3] _all_dbs and _dbs_info return a JSON array so they don't have an
obvious place to insert a last metadata row.
[4] For example they have a constraint that documents "a" and "z"
cannot both be in the database at the same time. But when iterating
it's possible that "a" was there at the start. Then by the end, "a"
was removed and "z" added, so both "a" and "z" would appear in the
emitted stream. Note that FoundationDB has APIs which exhibit the same
"relaxed" constrains:
https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Joan Touzet <wo...@apache.org>.
Hi Ilya, one question:
On 2020-04-23 18:27, Ilya Khlopotov wrote:
> Hello,
>
> I did an experiment and would like to share the results. So far I implemented only _all_dbs/cursor
>
> Here is how it works (I have only 2 databases)
> curl -u adm:pass "http://127.0.0.1:15984/_all_dbs/cursor?limit=1" | jq '.'
> {
> "items": [
> "_users"
> ],
> "completed": false,
> "bookmark": "AYNsAAAAA2gCbQAAAAIxMWEBaAJtAAAAATdtAAAAAf9oAm0AAAABNW0AAAAEdGVzdGo"
> }
>
> curl -u adm:pass "http://127.0.0.1:15984/_all_dbs/cursor?bookmark=AYNsAAAAA2gCbQAAAAIxMWEBaAJtAAAAATdtAAAAAf9oAm0AAAABNW0AAAAEdGVzdGo" | jq '.'
> {
> "items": [
> "test"
> ],
> "completed": true
> }
>
> The bookmark "remembers" all initial query parameters passed to the endpoint and it is opaque for end user.
>
> The completion field was tricky to obtain. The solution was to request `Limit + 1` elements from FDB. If FDB returns less than limit it means that there are no more elements and we can set `completed` to `true`.
Is this true if documents are each the max size (10MB)? What about with
Redwood?
I'm just worried that if the time to return a large set of documents
exceeds 5s (or whatever it is now), you might get back less than limit+1
docs and your logic would not work.
>
> Alternative names for the endpoint which came to mind:
> - page
> - cursor
> - step
> - iterate
> - continue
>
> During the experiment I learned that it is impossible to calculate the total and in such case maintaining `page` field wouldn't make sense.
>
> In regards to `update_seq` my thinking was that user would get information when the request was started and after finishing iteration she can request changes since that update_seq with a filter function to return only updated keys in the range. I agree that it is currently not used and we can it when we decide it is worth it.
>
> Best regards,
> iilyak
>
> On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
>> Hello everyone,
>>
>> Based on the discussions on the thread I would like to propose a number of first steps:
>> 1) introduce new endpoints
>> - {db}/_all_docs/page
>> - {db}/_all_docs/queries/page
>> - _all_dbs/page
>> - _dbs_info/page
>> - {db}/_design/{ddoc}/_view/{view}/page
>> - {db}/_design/{ddoc}/_view/{view}/queries/page
>> - {db}/_find/page
>>
>> These new endpoints would act as follows:
>> - don't use delayed responses
>> - return object with following structure
>> ```
>> {
>> "total": Total,
>> "bookmark": base64 encoded opaque value,
>> "completed": true | false,
>> "update_seq": when available,
>> "page": current page number,
>> "items": [
>> ]
>> }
>> ```
>> - the bookmark would include following data (base64 or protobuff???):
>> - direction
>> - page
>> - descending
>> - endkey
>> - endkey_docid
>> - inclusive_end
>> - startkey
>> - startkey_docid
>> - last_key
>> - update_seq
>> - timestamp
>> ```
>>
>> 2) Implement per-endpoint configurable max limits
>> ```
>> _all_docs = 5000
>> _all_docs/queries = 5000
>> _all_dbs = 5000
>> _dbs_info = 5000
>> _view = 2500
>> _view/queries = 2500
>> _find = 2500
>> ```
>>
>> Latter (after few years) CouchDB would deprecate and remove old endpoints.
>>
>> Best regards,
>> iilyak
>>
>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>>> Hello everyone,
>>>
>>> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>>>
>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>> read from the database. These are the endpoints I was thinking of:
>>>
>>> _all_docs, _all_dbs, _dbs_info and query results
>>>
>>> I want to focus on what happens when FoundationDB transactions
>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>> feeds, will crash or freeze. The reason is because the
>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>> default, so the request handlers run again and end up shoving the
>>> whole request down the socket again, headers and all, which is
>>> obviously broken and not what we want.
>>>
>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>> present some behaviors but feel free to add more. Some ideas might
>>> have been discounted on the IRC discussion already but I'll present
>>> them anyway in case is sparks further conversation:
>>>
>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>> streaming the data from the next key after last emitted in the
>>> previous transaction. Document the API behavior change that it may
>>> present a view of the data is never a point-in-time[4] snapshot of the
>>> DB.
>>>
>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>> - This is the easiest to implement since it would re-use the
>>> implementation for _changes feed (an extra option passed to the fold
>>> function).
>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>> snapshot view of the data.
>>>
>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>> param which indicates they are aware the stream may end early and so
>>> would have to paginate from the last emitted key with a skip=1. This
>>> will keep the request bodies the same as current CouchDB. However, if
>>> the users got all the data one request, they will end up wasting
>>> another request to see if there is more data available. If they didn't
>>> get any data they might have a too large of a skip value (see [2]) so
>>> would have to guess different values for start/end keys. Or impose max
>>> limit for the `skip` parameter.
>>>
>>> C) End the stream and add a final metadata row like a "transaction":
>>> "timeout" at the end. That will let the user know to keep paginating
>>> from the last key onward. This won't work for `_all_dbs` and
>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>> feeds and only use this for views and and _all_docs? If we like this
>>> choice, let's think what happens for those as I couldn't come up with
>>> anything decent there.
>>>
>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>> "key" of where the iteration stopped and the current "skip" and
>>> "limit" params, which would keep decreasing. Then user would pass
>>> those in "start_key=..." in the next request along with the limit and
>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>> "key":"..."}. This has the same issue with array results for
>>> `_all_dbs` and `_dbs_info`[3].
>>>
>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>> there such that response time is likely to fit in one transaction.
>>> This could be tricky as different runtime environments will have
>>> different characteristics. Also, if the timeout happens there isn't a
>>> a nice way to send an HTTP error since we already sent the 200
>>> response. The downside is that this might break how some users use the
>>> API, if say the are using large skips and limits already. Perhaps here
>>> we do both B and D, such that if users want transactional behavior,
>>> they specify that `transaction=true` param and only then we enforce
>>> low limit and skip maximums.
>>>
>>> F) At least for `_all_docs` it seems providing a point-in-time
>>> snapshot view doesn't necessarily need to be tied to transaction
>>> boundaries. We could check the update sequence of the database at the
>>> start of the next transaction and if it hasn't changed we can continue
>>> emitting a consistent view. This can apply to C and D and would just
>>> determine when the stream ends. If there are no writes happening to
>>> the db, this could potential streams all the data just like option A
>>> would do. Not entirely sure if this would work for views.
>>>
>>> So what do we think? I can see different combinations of options here,
>>> maybe even different for each API point. For example `_all_dbs`,
>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>> have parameters to do F, etc.
>>>
>>> Cheers,
>>> -Nick
>>>
>>> Some footnotes:
>>>
>>> [1] _changes feeds is the only one that works currently. It behaves as
>>> per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
>>> That is, we continue streaming the data by resetting the transaction
>>> object and restarting from the last emitted key (db sequence in this
>>> case). However, because the transaction restarts if a document is
>>> updated while the streaming take place, it may appear in the _changes
>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>> have to document it, since previously we presented this point-in-time
>>> snapshot of the database from when we started streaming.
>>>
>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>> currently support efficient offsets for key selectors
>>> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
>>> we implemented skip by iterating over the data. This means that a skip
>>> of say 100000 could keep timing out the transaction without yielding
>>> any data.
>>>
>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>> obvious place to insert a last metadata row.
>>>
>>> [4] For example they have a constraint that documents "a" and "z"
>>> cannot both be in the database at the same time. But when iterating
>>> it's possible that "a" was there at the start. Then by the end, "a"
>>> was removed and "z" added, so both "a" and "z" would appear in the
>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>> "relaxed" constrains:
>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>
>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
Hello,
I did an experiment and would like to share the results. So far I implemented only _all_dbs/cursor
Here is how it works (I have only 2 databases)
curl -u adm:pass "http://127.0.0.1:15984/_all_dbs/cursor?limit=1" | jq '.'
{
"items": [
"_users"
],
"completed": false,
"bookmark": "AYNsAAAAA2gCbQAAAAIxMWEBaAJtAAAAATdtAAAAAf9oAm0AAAABNW0AAAAEdGVzdGo"
}
curl -u adm:pass "http://127.0.0.1:15984/_all_dbs/cursor?bookmark=AYNsAAAAA2gCbQAAAAIxMWEBaAJtAAAAATdtAAAAAf9oAm0AAAABNW0AAAAEdGVzdGo" | jq '.'
{
"items": [
"test"
],
"completed": true
}
The bookmark "remembers" all initial query parameters passed to the endpoint and it is opaque for end user.
The completion field was tricky to obtain. The solution was to request `Limit + 1` elements from FDB. If FDB returns less than limit it means that there are no more elements and we can set `completed` to `true`.
Alternative names for the endpoint which came to mind:
- page
- cursor
- step
- iterate
- continue
During the experiment I learned that it is impossible to calculate the total and in such case maintaining `page` field wouldn't make sense.
In regards to `update_seq` my thinking was that user would get information when the request was started and after finishing iteration she can request changes since that update_seq with a filter function to return only updated keys in the range. I agree that it is currently not used and we can it when we decide it is worth it.
Best regards,
iilyak
On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
> Hello everyone,
>
> Based on the discussions on the thread I would like to propose a number of first steps:
> 1) introduce new endpoints
> - {db}/_all_docs/page
> - {db}/_all_docs/queries/page
> - _all_dbs/page
> - _dbs_info/page
> - {db}/_design/{ddoc}/_view/{view}/page
> - {db}/_design/{ddoc}/_view/{view}/queries/page
> - {db}/_find/page
>
> These new endpoints would act as follows:
> - don't use delayed responses
> - return object with following structure
> ```
> {
> "total": Total,
> "bookmark": base64 encoded opaque value,
> "completed": true | false,
> "update_seq": when available,
> "page": current page number,
> "items": [
> ]
> }
> ```
> - the bookmark would include following data (base64 or protobuff???):
> - direction
> - page
> - descending
> - endkey
> - endkey_docid
> - inclusive_end
> - startkey
> - startkey_docid
> - last_key
> - update_seq
> - timestamp
> ```
>
> 2) Implement per-endpoint configurable max limits
> ```
> _all_docs = 5000
> _all_docs/queries = 5000
> _all_dbs = 5000
> _dbs_info = 5000
> _view = 2500
> _view/queries = 2500
> _find = 2500
> ```
>
> Latter (after few years) CouchDB would deprecate and remove old endpoints.
>
> Best regards,
> iilyak
>
> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > Hello everyone,
> >
> > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> >
> > By "streaming APIs" I mean APIs which stream data in row as it gets
> > read from the database. These are the endpoints I was thinking of:
> >
> > _all_docs, _all_dbs, _dbs_info and query results
> >
> > I want to focus on what happens when FoundationDB transactions
> > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > feeds, will crash or freeze. The reason is because the
> > transaction_too_old error at the end of 5 seconds is retry-able by
> > default, so the request handlers run again and end up shoving the
> > whole request down the socket again, headers and all, which is
> > obviously broken and not what we want.
> >
> > There are few alternatives discussed in couchdb-dev channel. I'll
> > present some behaviors but feel free to add more. Some ideas might
> > have been discounted on the IRC discussion already but I'll present
> > them anyway in case is sparks further conversation:
> >
> > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > streaming the data from the next key after last emitted in the
> > previous transaction. Document the API behavior change that it may
> > present a view of the data is never a point-in-time[4] snapshot of the
> > DB.
> >
> > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > don't have to change to continue using these CouchDB 4.0 endpoints
> > - This is the easiest to implement since it would re-use the
> > implementation for _changes feed (an extra option passed to the fold
> > function).
> > - Breaks API behavior if users relied on having a point-in-time[4]
> > snapshot view of the data.
> >
> > B) Simply end the stream. Let the users pass a `?transaction=true`
> > param which indicates they are aware the stream may end early and so
> > would have to paginate from the last emitted key with a skip=1. This
> > will keep the request bodies the same as current CouchDB. However, if
> > the users got all the data one request, they will end up wasting
> > another request to see if there is more data available. If they didn't
> > get any data they might have a too large of a skip value (see [2]) so
> > would have to guess different values for start/end keys. Or impose max
> > limit for the `skip` parameter.
> >
> > C) End the stream and add a final metadata row like a "transaction":
> > "timeout" at the end. That will let the user know to keep paginating
> > from the last key onward. This won't work for `_all_dbs` and
> > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > feeds and only use this for views and and _all_docs? If we like this
> > choice, let's think what happens for those as I couldn't come up with
> > anything decent there.
> >
> > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > "key" of where the iteration stopped and the current "skip" and
> > "limit" params, which would keep decreasing. Then user would pass
> > those in "start_key=..." in the next request along with the limit and
> > skip params. So something like "continuation":{"skip":599, "limit":5,
> > "key":"..."}. This has the same issue with array results for
> > `_all_dbs` and `_dbs_info`[3].
> >
> > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > there such that response time is likely to fit in one transaction.
> > This could be tricky as different runtime environments will have
> > different characteristics. Also, if the timeout happens there isn't a
> > a nice way to send an HTTP error since we already sent the 200
> > response. The downside is that this might break how some users use the
> > API, if say the are using large skips and limits already. Perhaps here
> > we do both B and D, such that if users want transactional behavior,
> > they specify that `transaction=true` param and only then we enforce
> > low limit and skip maximums.
> >
> > F) At least for `_all_docs` it seems providing a point-in-time
> > snapshot view doesn't necessarily need to be tied to transaction
> > boundaries. We could check the update sequence of the database at the
> > start of the next transaction and if it hasn't changed we can continue
> > emitting a consistent view. This can apply to C and D and would just
> > determine when the stream ends. If there are no writes happening to
> > the db, this could potential streams all the data just like option A
> > would do. Not entirely sure if this would work for views.
> >
> > So what do we think? I can see different combinations of options here,
> > maybe even different for each API point. For example `_all_dbs`,
> > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > have parameters to do F, etc.
> >
> > Cheers,
> > -Nick
> >
> > Some footnotes:
> >
> > [1] _changes feeds is the only one that works currently. It behaves as
> > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > That is, we continue streaming the data by resetting the transaction
> > object and restarting from the last emitted key (db sequence in this
> > case). However, because the transaction restarts if a document is
> > updated while the streaming take place, it may appear in the _changes
> > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > have to document it, since previously we presented this point-in-time
> > snapshot of the database from when we started streaming.
> >
> > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > currently support efficient offsets for key selectors
> > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > we implemented skip by iterating over the data. This means that a skip
> > of say 100000 could keep timing out the transaction without yielding
> > any data.
> >
> > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > obvious place to insert a last metadata row.
> >
> > [4] For example they have a constraint that documents "a" and "z"
> > cannot both be in the database at the same time. But when iterating
> > it's possible that "a" was there at the start. Then by the end, "a"
> > was removed and "z" added, so both "a" and "z" would appear in the
> > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > "relaxed" constrains:
> > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Ilya,
Initial reaction — there’s a lot to like here. It seems like a pragmatic step forward for the current API that handles the corner case of large responses while maintaining compatibility for the large majority of API requests that don’t exceed this limit.
I think the `limit` parameter might be overloaded, though. If we add this pagination capability then `limit` could be used to refer to the size of a “page” or to the size of the overall result set. In the current proposal there’s no easy way to say “give me 5,000 results in blocks of 100”. Maybe it makes sense to add `page_size` as a new parameter and keep `limit` as the total number of results returned (i.e., once the `limit` is reached the final response will not have a “next” bookmark).
If we do introduce a `page_size` I would not impose a maximum on `limit`. It should be possible to use bookmarks to page through an entire database.
To your other questions:
> - `bookmark` vs `token`?
Prefer `bookmark` since that’s already used for _find and _search
> - should we prohibit setting other fields when bookmark is set?
As a general rule, yes. I can see an exception around allowing users to say whether they want to retrieve the next page of results using the same FDB read version as the previous request, and how the server should proceed if that read version is now too old. But that’s for later.
> - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
I think I would avoid including the scheme and server. I could see including the full path instead of just the bookmark, particularly as you’re including the “first” bit (which is a nice touch).
A couple of other comments:
> Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
I think you mean `_all_dbs` instead of `_all_docs` here, and I agree that’s fine to defer.
> - don't use delayed responses when `bookmark` field is provided
> - don't use delayed responses when `limit` query key is specified and when it is below the max limit
> - return 400 when limit query key is specified and it is greater than the max limit
> - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
If we add the `page_size` parameter does this become
- don't use delayed responses when `bookmark` field is provided
- don't use delayed responses when `page_size` query key is specified and when it is below the max
- return 400 when `page_size` query key is specified and it is greater than the max
I feel like the discussion of using delayed response or not is just a performance optimization, assuming we’re able to identify the right status code before streaming. So it could be omitted.
Choosing a default `page_size` requires some thought. I think we want to have a default, even though it is a behavior change. At the same time maybe we can minimize the number of times where pagination needs to show up with something like the following:
- user supplies `limit` less than max page size -> set `page_size` = `limit`
- user omits `limit`or sets one that exceeds max page size -> set `page_size`= ~100
Regards, Adam
> On Apr 28, 2020, at 7:56 AM, Ilya Khlopotov <ii...@apache.org> wrote:
>
> Hello,
>
> I would like to introduce second proposal.
>
> 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
> - {db}/_all_docs
> - {db}/_all_docs/queries
> - _dbs_info
> - {db}/_design/{ddoc}/_view/{view}
> - {db}/_design/{ddoc}/_view/{view}/queries
> 2) Add following additional fields into response:
> ```
> "first": {
> "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
> },
> "previous": {
> "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
> },
> "next": {
> "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
> },
> ```
> 3) Implement per-endpoint configurable max limits
> ```
> [request_limits]
> _all_docs = 5000
> _all_docs/queries = 5000
> _all_dbs = 5000
> _dbs_info = 5000
> _view = 2500
> _view/queries = 2500
> _find = 2500
> ```
> 4) Implement following semantics:
> - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
> - don't use delayed responses when `bookmark` field is provided
> - don't use delayed responses when `limit` query key is specified and when it is below the max limit
> - return 400 when limit query key is specified and it is greater than the max limit
> - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
> - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
>
> Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
>
> Questions:
> - `bookmark` vs `token`?
> - should we prohibit setting other fields when bookmark is set?
> - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
>
> Best regards,
> iilyak
>
> On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
>> Hello everyone,
>>
>> Based on the discussions on the thread I would like to propose a number of first steps:
>> 1) introduce new endpoints
>> - {db}/_all_docs/page
>> - {db}/_all_docs/queries/page
>> - _all_dbs/page
>> - _dbs_info/page
>> - {db}/_design/{ddoc}/_view/{view}/page
>> - {db}/_design/{ddoc}/_view/{view}/queries/page
>> - {db}/_find/page
>>
>> These new endpoints would act as follows:
>> - don't use delayed responses
>> - return object with following structure
>> ```
>> {
>> "total": Total,
>> "bookmark": base64 encoded opaque value,
>> "completed": true | false,
>> "update_seq": when available,
>> "page": current page number,
>> "items": [
>> ]
>> }
>> ```
>> - the bookmark would include following data (base64 or protobuff???):
>> - direction
>> - page
>> - descending
>> - endkey
>> - endkey_docid
>> - inclusive_end
>> - startkey
>> - startkey_docid
>> - last_key
>> - update_seq
>> - timestamp
>> ```
>>
>> 2) Implement per-endpoint configurable max limits
>> ```
>> _all_docs = 5000
>> _all_docs/queries = 5000
>> _all_dbs = 5000
>> _dbs_info = 5000
>> _view = 2500
>> _view/queries = 2500
>> _find = 2500
>> ```
>>
>> Latter (after few years) CouchDB would deprecate and remove old endpoints.
>>
>> Best regards,
>> iilyak
>>
>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>>> Hello everyone,
>>>
>>> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>>>
>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>> read from the database. These are the endpoints I was thinking of:
>>>
>>> _all_docs, _all_dbs, _dbs_info and query results
>>>
>>> I want to focus on what happens when FoundationDB transactions
>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>> feeds, will crash or freeze. The reason is because the
>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>> default, so the request handlers run again and end up shoving the
>>> whole request down the socket again, headers and all, which is
>>> obviously broken and not what we want.
>>>
>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>> present some behaviors but feel free to add more. Some ideas might
>>> have been discounted on the IRC discussion already but I'll present
>>> them anyway in case is sparks further conversation:
>>>
>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>> streaming the data from the next key after last emitted in the
>>> previous transaction. Document the API behavior change that it may
>>> present a view of the data is never a point-in-time[4] snapshot of the
>>> DB.
>>>
>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>> - This is the easiest to implement since it would re-use the
>>> implementation for _changes feed (an extra option passed to the fold
>>> function).
>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>> snapshot view of the data.
>>>
>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>> param which indicates they are aware the stream may end early and so
>>> would have to paginate from the last emitted key with a skip=1. This
>>> will keep the request bodies the same as current CouchDB. However, if
>>> the users got all the data one request, they will end up wasting
>>> another request to see if there is more data available. If they didn't
>>> get any data they might have a too large of a skip value (see [2]) so
>>> would have to guess different values for start/end keys. Or impose max
>>> limit for the `skip` parameter.
>>>
>>> C) End the stream and add a final metadata row like a "transaction":
>>> "timeout" at the end. That will let the user know to keep paginating
>>> from the last key onward. This won't work for `_all_dbs` and
>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>> feeds and only use this for views and and _all_docs? If we like this
>>> choice, let's think what happens for those as I couldn't come up with
>>> anything decent there.
>>>
>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>> "key" of where the iteration stopped and the current "skip" and
>>> "limit" params, which would keep decreasing. Then user would pass
>>> those in "start_key=..." in the next request along with the limit and
>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>> "key":"..."}. This has the same issue with array results for
>>> `_all_dbs` and `_dbs_info`[3].
>>>
>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>> there such that response time is likely to fit in one transaction.
>>> This could be tricky as different runtime environments will have
>>> different characteristics. Also, if the timeout happens there isn't a
>>> a nice way to send an HTTP error since we already sent the 200
>>> response. The downside is that this might break how some users use the
>>> API, if say the are using large skips and limits already. Perhaps here
>>> we do both B and D, such that if users want transactional behavior,
>>> they specify that `transaction=true` param and only then we enforce
>>> low limit and skip maximums.
>>>
>>> F) At least for `_all_docs` it seems providing a point-in-time
>>> snapshot view doesn't necessarily need to be tied to transaction
>>> boundaries. We could check the update sequence of the database at the
>>> start of the next transaction and if it hasn't changed we can continue
>>> emitting a consistent view. This can apply to C and D and would just
>>> determine when the stream ends. If there are no writes happening to
>>> the db, this could potential streams all the data just like option A
>>> would do. Not entirely sure if this would work for views.
>>>
>>> So what do we think? I can see different combinations of options here,
>>> maybe even different for each API point. For example `_all_dbs`,
>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>> have parameters to do F, etc.
>>>
>>> Cheers,
>>> -Nick
>>>
>>> Some footnotes:
>>>
>>> [1] _changes feeds is the only one that works currently. It behaves as
>>> per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
>>> That is, we continue streaming the data by resetting the transaction
>>> object and restarting from the last emitted key (db sequence in this
>>> case). However, because the transaction restarts if a document is
>>> updated while the streaming take place, it may appear in the _changes
>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>> have to document it, since previously we presented this point-in-time
>>> snapshot of the database from when we started streaming.
>>>
>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>> currently support efficient offsets for key selectors
>>> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
>>> we implemented skip by iterating over the data. This means that a skip
>>> of say 100000 could keep timing out the transaction without yielding
>>> any data.
>>>
>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>> obvious place to insert a last metadata row.
>>>
>>> [4] For example they have a constraint that documents "a" and "z"
>>> cannot both be in the database at the same time. But when iterating
>>> it's possible that "a" was there at the start. Then by the end, "a"
>>> was removed and "z" added, so both "a" and "z" would appear in the
>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>> "relaxed" constrains:
>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>
>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Adam Kocoloski <ko...@apache.org>.
I think this is a good reason to fall back to just including the value of the bookmark in “first”, “next” and “last”, and then leaving it up to the client to decide whether to supply the bookmark in the URL or in the request body.
Adam
> On Apr 30, 2020, at 10:47 AM, Joan Touzet <wo...@apache.org> wrote:
>
> Can we keep a distributed hash of doc keys server side with a smaller handle we hand to clients?
>
> If the cache can't be found or a restart happens, oh well.
>
> -Joan
>
> On 2020-04-30 10:23, Ilya Khlopotov wrote:
>> There is a problem with representing `next`/`previous`/`first` as path. With 5kB sized doc keys and we could exceed max URL length (8192 bytes). This means we would have to support POST. The question is how to handle the case when the URL is greater than 8192. The problem is CouchDB don't knows it's DNS name so we don't know the safe value to compare with.
>> Options are:
>> 1) always use POST for pagination
>> 2) add server_name to config and return error when bookmark length exceeds dynamically calculated threshold. The threshold would account for db name length, server_name length, port and scheme length.
>> - what error to return?
>> I think option number 2 is too subtle to implement.
>> The downside of option 1 is it is a bit harder to use from the browser or curl.
>> On 2020/04/29 17:27:59, Ilya Khlopotov <ii...@apache.org> wrote:
>>> I think I addressed all comments and created an RFC https://github.com/apache/couchdb-documentation/pull/530
>>>
>>> On 2020/04/28 11:56:15, Ilya Khlopotov <ii...@apache.org> wrote:
>>>> Hello,
>>>>
>>>> I would like to introduce second proposal.
>>>>
>>>> 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
>>>> - {db}/_all_docs
>>>> - {db}/_all_docs/queries
>>>> - _dbs_info
>>>> - {db}/_design/{ddoc}/_view/{view}
>>>> - {db}/_design/{ddoc}/_view/{view}/queries
>>>> 2) Add following additional fields into response:
>>>> ```
>>>> "first": {
>>>> "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
>>>> },
>>>> "previous": {
>>>> "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
>>>> },
>>>> "next": {
>>>> "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
>>>> },
>>>> ```
>>>> 3) Implement per-endpoint configurable max limits
>>>> ```
>>>> [request_limits]
>>>> _all_docs = 5000
>>>> _all_docs/queries = 5000
>>>> _all_dbs = 5000
>>>> _dbs_info = 5000
>>>> _view = 2500
>>>> _view/queries = 2500
>>>> _find = 2500
>>>> ```
>>>> 4) Implement following semantics:
>>>> - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
>>>> - don't use delayed responses when `bookmark` field is provided
>>>> - don't use delayed responses when `limit` query key is specified and when it is below the max limit
>>>> - return 400 when limit query key is specified and it is greater than the max limit
>>>> - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
>>>> - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
>>>>
>>>> Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
>>>> Questions:
>>>> - `bookmark` vs `token`?
>>>> - should we prohibit setting other fields when bookmark is set?
>>>> - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
>>>>
>>>> Best regards,
>>>> iilyak
>>>>
>>>> On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
>>>>> Hello everyone,
>>>>>
>>>>> Based on the discussions on the thread I would like to propose a number of first steps:
>>>>> 1) introduce new endpoints
>>>>> - {db}/_all_docs/page
>>>>> - {db}/_all_docs/queries/page
>>>>> - _all_dbs/page
>>>>> - _dbs_info/page
>>>>> - {db}/_design/{ddoc}/_view/{view}/page
>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
>>>>> - {db}/_find/page
>>>>>
>>>>> These new endpoints would act as follows:
>>>>> - don't use delayed responses
>>>>> - return object with following structure
>>>>> ```
>>>>> {
>>>>> "total": Total,
>>>>> "bookmark": base64 encoded opaque value,
>>>>> "completed": true | false,
>>>>> "update_seq": when available,
>>>>> "page": current page number,
>>>>> "items": [
>>>>> ]
>>>>> }
>>>>> ```
>>>>> - the bookmark would include following data (base64 or protobuff???):
>>>>> - direction
>>>>> - page
>>>>> - descending
>>>>> - endkey
>>>>> - endkey_docid
>>>>> - inclusive_end
>>>>> - startkey
>>>>> - startkey_docid
>>>>> - last_key
>>>>> - update_seq
>>>>> - timestamp
>>>>> ```
>>>>>
>>>>> 2) Implement per-endpoint configurable max limits
>>>>> ```
>>>>> _all_docs = 5000
>>>>> _all_docs/queries = 5000
>>>>> _all_dbs = 5000
>>>>> _dbs_info = 5000
>>>>> _view = 2500
>>>>> _view/queries = 2500
>>>>> _find = 2500
>>>>> ```
>>>>>
>>>>> Latter (after few years) CouchDB would deprecate and remove old endpoints.
>>>>>
>>>>> Best regards,
>>>>> iilyak
>>>>>
>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>>>>>> Hello everyone,
>>>>>>
>>>>>> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>>>>>>
>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>>>>> read from the database. These are the endpoints I was thinking of:
>>>>>>
>>>>>> _all_docs, _all_dbs, _dbs_info and query results
>>>>>>
>>>>>> I want to focus on what happens when FoundationDB transactions
>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>>>>> feeds, will crash or freeze. The reason is because the
>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>>>>> default, so the request handlers run again and end up shoving the
>>>>>> whole request down the socket again, headers and all, which is
>>>>>> obviously broken and not what we want.
>>>>>>
>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>>>>> present some behaviors but feel free to add more. Some ideas might
>>>>>> have been discounted on the IRC discussion already but I'll present
>>>>>> them anyway in case is sparks further conversation:
>>>>>>
>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>>>>> streaming the data from the next key after last emitted in the
>>>>>> previous transaction. Document the API behavior change that it may
>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
>>>>>> DB.
>>>>>>
>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>>>>> - This is the easiest to implement since it would re-use the
>>>>>> implementation for _changes feed (an extra option passed to the fold
>>>>>> function).
>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>>>>> snapshot view of the data.
>>>>>>
>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>>>>> param which indicates they are aware the stream may end early and so
>>>>>> would have to paginate from the last emitted key with a skip=1. This
>>>>>> will keep the request bodies the same as current CouchDB. However, if
>>>>>> the users got all the data one request, they will end up wasting
>>>>>> another request to see if there is more data available. If they didn't
>>>>>> get any data they might have a too large of a skip value (see [2]) so
>>>>>> would have to guess different values for start/end keys. Or impose max
>>>>>> limit for the `skip` parameter.
>>>>>>
>>>>>> C) End the stream and add a final metadata row like a "transaction":
>>>>>> "timeout" at the end. That will let the user know to keep paginating
>>>>>> from the last key onward. This won't work for `_all_dbs` and
>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>>>>> feeds and only use this for views and and _all_docs? If we like this
>>>>>> choice, let's think what happens for those as I couldn't come up with
>>>>>> anything decent there.
>>>>>>
>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>>>>> "key" of where the iteration stopped and the current "skip" and
>>>>>> "limit" params, which would keep decreasing. Then user would pass
>>>>>> those in "start_key=..." in the next request along with the limit and
>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>>>>> "key":"..."}. This has the same issue with array results for
>>>>>> `_all_dbs` and `_dbs_info`[3].
>>>>>>
>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>>>>> there such that response time is likely to fit in one transaction.
>>>>>> This could be tricky as different runtime environments will have
>>>>>> different characteristics. Also, if the timeout happens there isn't a
>>>>>> a nice way to send an HTTP error since we already sent the 200
>>>>>> response. The downside is that this might break how some users use the
>>>>>> API, if say the are using large skips and limits already. Perhaps here
>>>>>> we do both B and D, such that if users want transactional behavior,
>>>>>> they specify that `transaction=true` param and only then we enforce
>>>>>> low limit and skip maximums.
>>>>>>
>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
>>>>>> snapshot view doesn't necessarily need to be tied to transaction
>>>>>> boundaries. We could check the update sequence of the database at the
>>>>>> start of the next transaction and if it hasn't changed we can continue
>>>>>> emitting a consistent view. This can apply to C and D and would just
>>>>>> determine when the stream ends. If there are no writes happening to
>>>>>> the db, this could potential streams all the data just like option A
>>>>>> would do. Not entirely sure if this would work for views.
>>>>>>
>>>>>> So what do we think? I can see different combinations of options here,
>>>>>> maybe even different for each API point. For example `_all_dbs`,
>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>>>>> have parameters to do F, etc.
>>>>>>
>>>>>> Cheers,
>>>>>> -Nick
>>>>>>
>>>>>> Some footnotes:
>>>>>>
>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
>>>>>> per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
>>>>>> That is, we continue streaming the data by resetting the transaction
>>>>>> object and restarting from the last emitted key (db sequence in this
>>>>>> case). However, because the transaction restarts if a document is
>>>>>> updated while the streaming take place, it may appear in the _changes
>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>>>>> have to document it, since previously we presented this point-in-time
>>>>>> snapshot of the database from when we started streaming.
>>>>>>
>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>>>>> currently support efficient offsets for key selectors
>>>>>> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
>>>>>> we implemented skip by iterating over the data. This means that a skip
>>>>>> of say 100000 could keep timing out the transaction without yielding
>>>>>> any data.
>>>>>>
>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>>>>> obvious place to insert a last metadata row.
>>>>>>
>>>>>> [4] For example they have a constraint that documents "a" and "z"
>>>>>> cannot both be in the database at the same time. But when iterating
>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>>>>> "relaxed" constrains:
>>>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>>>>
>>>>>
>>>>
>>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Joan Touzet <wo...@apache.org>.
Can we keep a distributed hash of doc keys server side with a smaller
handle we hand to clients?
If the cache can't be found or a restart happens, oh well.
-Joan
On 2020-04-30 10:23, Ilya Khlopotov wrote:
> There is a problem with representing `next`/`previous`/`first` as path. With 5kB sized doc keys and we could exceed max URL length (8192 bytes). This means we would have to support POST. The question is how to handle the case when the URL is greater than 8192. The problem is CouchDB don't knows it's DNS name so we don't know the safe value to compare with.
>
> Options are:
> 1) always use POST for pagination
> 2) add server_name to config and return error when bookmark length exceeds dynamically calculated threshold. The threshold would account for db name length, server_name length, port and scheme length.
> - what error to return?
>
> I think option number 2 is too subtle to implement.
>
> The downside of option 1 is it is a bit harder to use from the browser or curl.
>
> On 2020/04/29 17:27:59, Ilya Khlopotov <ii...@apache.org> wrote:
>> I think I addressed all comments and created an RFC https://github.com/apache/couchdb-documentation/pull/530
>>
>> On 2020/04/28 11:56:15, Ilya Khlopotov <ii...@apache.org> wrote:
>>> Hello,
>>>
>>> I would like to introduce second proposal.
>>>
>>> 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
>>> - {db}/_all_docs
>>> - {db}/_all_docs/queries
>>> - _dbs_info
>>> - {db}/_design/{ddoc}/_view/{view}
>>> - {db}/_design/{ddoc}/_view/{view}/queries
>>> 2) Add following additional fields into response:
>>> ```
>>> "first": {
>>> "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
>>> },
>>> "previous": {
>>> "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
>>> },
>>> "next": {
>>> "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
>>> },
>>> ```
>>> 3) Implement per-endpoint configurable max limits
>>> ```
>>> [request_limits]
>>> _all_docs = 5000
>>> _all_docs/queries = 5000
>>> _all_dbs = 5000
>>> _dbs_info = 5000
>>> _view = 2500
>>> _view/queries = 2500
>>> _find = 2500
>>> ```
>>> 4) Implement following semantics:
>>> - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
>>> - don't use delayed responses when `bookmark` field is provided
>>> - don't use delayed responses when `limit` query key is specified and when it is below the max limit
>>> - return 400 when limit query key is specified and it is greater than the max limit
>>> - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
>>> - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
>>>
>>> Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
>>>
>>> Questions:
>>> - `bookmark` vs `token`?
>>> - should we prohibit setting other fields when bookmark is set?
>>> - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
>>>
>>> Best regards,
>>> iilyak
>>>
>>> On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
>>>> Hello everyone,
>>>>
>>>> Based on the discussions on the thread I would like to propose a number of first steps:
>>>> 1) introduce new endpoints
>>>> - {db}/_all_docs/page
>>>> - {db}/_all_docs/queries/page
>>>> - _all_dbs/page
>>>> - _dbs_info/page
>>>> - {db}/_design/{ddoc}/_view/{view}/page
>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
>>>> - {db}/_find/page
>>>>
>>>> These new endpoints would act as follows:
>>>> - don't use delayed responses
>>>> - return object with following structure
>>>> ```
>>>> {
>>>> "total": Total,
>>>> "bookmark": base64 encoded opaque value,
>>>> "completed": true | false,
>>>> "update_seq": when available,
>>>> "page": current page number,
>>>> "items": [
>>>> ]
>>>> }
>>>> ```
>>>> - the bookmark would include following data (base64 or protobuff???):
>>>> - direction
>>>> - page
>>>> - descending
>>>> - endkey
>>>> - endkey_docid
>>>> - inclusive_end
>>>> - startkey
>>>> - startkey_docid
>>>> - last_key
>>>> - update_seq
>>>> - timestamp
>>>> ```
>>>>
>>>> 2) Implement per-endpoint configurable max limits
>>>> ```
>>>> _all_docs = 5000
>>>> _all_docs/queries = 5000
>>>> _all_dbs = 5000
>>>> _dbs_info = 5000
>>>> _view = 2500
>>>> _view/queries = 2500
>>>> _find = 2500
>>>> ```
>>>>
>>>> Latter (after few years) CouchDB would deprecate and remove old endpoints.
>>>>
>>>> Best regards,
>>>> iilyak
>>>>
>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>>>>> Hello everyone,
>>>>>
>>>>> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>>>>>
>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>>>> read from the database. These are the endpoints I was thinking of:
>>>>>
>>>>> _all_docs, _all_dbs, _dbs_info and query results
>>>>>
>>>>> I want to focus on what happens when FoundationDB transactions
>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>>>> feeds, will crash or freeze. The reason is because the
>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>>>> default, so the request handlers run again and end up shoving the
>>>>> whole request down the socket again, headers and all, which is
>>>>> obviously broken and not what we want.
>>>>>
>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>>>> present some behaviors but feel free to add more. Some ideas might
>>>>> have been discounted on the IRC discussion already but I'll present
>>>>> them anyway in case is sparks further conversation:
>>>>>
>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>>>> streaming the data from the next key after last emitted in the
>>>>> previous transaction. Document the API behavior change that it may
>>>>> present a view of the data is never a point-in-time[4] snapshot of the
>>>>> DB.
>>>>>
>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>>>> - This is the easiest to implement since it would re-use the
>>>>> implementation for _changes feed (an extra option passed to the fold
>>>>> function).
>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>>>> snapshot view of the data.
>>>>>
>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>>>> param which indicates they are aware the stream may end early and so
>>>>> would have to paginate from the last emitted key with a skip=1. This
>>>>> will keep the request bodies the same as current CouchDB. However, if
>>>>> the users got all the data one request, they will end up wasting
>>>>> another request to see if there is more data available. If they didn't
>>>>> get any data they might have a too large of a skip value (see [2]) so
>>>>> would have to guess different values for start/end keys. Or impose max
>>>>> limit for the `skip` parameter.
>>>>>
>>>>> C) End the stream and add a final metadata row like a "transaction":
>>>>> "timeout" at the end. That will let the user know to keep paginating
>>>>> from the last key onward. This won't work for `_all_dbs` and
>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>>>> feeds and only use this for views and and _all_docs? If we like this
>>>>> choice, let's think what happens for those as I couldn't come up with
>>>>> anything decent there.
>>>>>
>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>>>> "key" of where the iteration stopped and the current "skip" and
>>>>> "limit" params, which would keep decreasing. Then user would pass
>>>>> those in "start_key=..." in the next request along with the limit and
>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>>>> "key":"..."}. This has the same issue with array results for
>>>>> `_all_dbs` and `_dbs_info`[3].
>>>>>
>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>>>> there such that response time is likely to fit in one transaction.
>>>>> This could be tricky as different runtime environments will have
>>>>> different characteristics. Also, if the timeout happens there isn't a
>>>>> a nice way to send an HTTP error since we already sent the 200
>>>>> response. The downside is that this might break how some users use the
>>>>> API, if say the are using large skips and limits already. Perhaps here
>>>>> we do both B and D, such that if users want transactional behavior,
>>>>> they specify that `transaction=true` param and only then we enforce
>>>>> low limit and skip maximums.
>>>>>
>>>>> F) At least for `_all_docs` it seems providing a point-in-time
>>>>> snapshot view doesn't necessarily need to be tied to transaction
>>>>> boundaries. We could check the update sequence of the database at the
>>>>> start of the next transaction and if it hasn't changed we can continue
>>>>> emitting a consistent view. This can apply to C and D and would just
>>>>> determine when the stream ends. If there are no writes happening to
>>>>> the db, this could potential streams all the data just like option A
>>>>> would do. Not entirely sure if this would work for views.
>>>>>
>>>>> So what do we think? I can see different combinations of options here,
>>>>> maybe even different for each API point. For example `_all_dbs`,
>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>>>> have parameters to do F, etc.
>>>>>
>>>>> Cheers,
>>>>> -Nick
>>>>>
>>>>> Some footnotes:
>>>>>
>>>>> [1] _changes feeds is the only one that works currently. It behaves as
>>>>> per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
>>>>> That is, we continue streaming the data by resetting the transaction
>>>>> object and restarting from the last emitted key (db sequence in this
>>>>> case). However, because the transaction restarts if a document is
>>>>> updated while the streaming take place, it may appear in the _changes
>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>>>> have to document it, since previously we presented this point-in-time
>>>>> snapshot of the database from when we started streaming.
>>>>>
>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>>>> currently support efficient offsets for key selectors
>>>>> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
>>>>> we implemented skip by iterating over the data. This means that a skip
>>>>> of say 100000 could keep timing out the transaction without yielding
>>>>> any data.
>>>>>
>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>>>> obvious place to insert a last metadata row.
>>>>>
>>>>> [4] For example they have a constraint that documents "a" and "z"
>>>>> cannot both be in the database at the same time. But when iterating
>>>>> it's possible that "a" was there at the start. Then by the end, "a"
>>>>> was removed and "z" added, so both "a" and "z" would appear in the
>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>>>> "relaxed" constrains:
>>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
> Maybe we should set a hard limit on the maximum doc ids size, 2-4KB?
> We have a config setting to do it already.
I am +100 for stricter limit. We need to limit db name and doc ids. However I do agree that it is a bigger change and requires more discussion.
> Also was curious, in the latest proposal what will be included in the
> bookmark? Proposal said "The bookmark would include information needed
> to ensure proper pagination without the need to repeat initial
> parameters of the request".
The main idea of the proposal is to return a bookmark which can be used to request next page without specifying any additional parameters. Even more we would forbid setting any extra parameters when bookmark is present. Which means that it has to include all parameters passed to initial request. In the PoC I am working on I took an approach where I encode all non default fields of mrargs record. However this is an implementation detail.
There is an interesting aspect of this approach which worth mentioning. In case when user pass the keys parameter we would include it into bookmark. However on a closer look this is not worse than what we have now. Currently if user wants to retrieve lots of documents with specified keys they have to use POST request and pass keys in each request. With bookmark support they would continue to use POST but they wouldn't need to provide the list of keys in each request because it would be encoded in the bookmark. We even can do a small optimization and remove already returned keys when we update bookmark to point to next page. Also bookmark is a compressed term.
Best regards,
iilyak
On 2020/04/30 17:35:18, Nick Vatamaniuc <va...@gmail.com> wrote:
> Hi Ilya,
>
> Maybe we should set a hard limit on the maximum doc ids size, 2-4KB?
> We have a config setting to do it already. And we also have hard
> limit of 10KB for FDB keys. Due to a limitation in Erlang http header
> parser, used through mochiweb, 8KB is (was?) the limit based on the
> default socket receive buffer size (see [1] for gory details).
>
> I was going to suggest 4KB initially but say users would want to
> specify a start_key and end_key + the dbname, so maybe 3KB or 2KB is a
> better option? It will be an incompatible change so that's something
> balance against.
>
> Regarding setting server_name in the config, you're referring to the
> specific limitation of 8KB from the mochweb header parsing, that
> shouldn't include the domain name, just individual request + header
> lines. It would look like "GET /dbname/_all_docs?bookmark...".
>
> Also was curious, in the latest proposal what will be included in the
> bookmark? Proposal said "The bookmark would include information needed
> to ensure proper pagination without the need to repeat initial
> parameters of the request".
>
> [1] https://github.com/apache/couchdb/commit/d23025ebd7176f6c307ddf49902cf20b33bd55c4
>
>
> On Thu, Apr 30, 2020 at 10:23 AM Ilya Khlopotov <ii...@apache.org> wrote:
> >
> > There is a problem with representing `next`/`previous`/`first` as path. With 5kB sized doc keys and we could exceed max URL length (8192 bytes). This means we would have to support POST. The question is how to handle the case when the URL is greater than 8192. The problem is CouchDB don't knows it's DNS name so we don't know the safe value to compare with.
> >
> > Options are:
> > 1) always use POST for pagination
> > 2) add server_name to config and return error when bookmark length exceeds dynamically calculated threshold. The threshold would account for db name length, server_name length, port and scheme length.
> > - what error to return?
> >
> > I think option number 2 is too subtle to implement.
> >
> > The downside of option 1 is it is a bit harder to use from the browser or curl.
> >
> > On 2020/04/29 17:27:59, Ilya Khlopotov <ii...@apache.org> wrote:
> > > I think I addressed all comments and created an RFC https://github.com/apache/couchdb-documentation/pull/530
> > >
> > > On 2020/04/28 11:56:15, Ilya Khlopotov <ii...@apache.org> wrote:
> > > > Hello,
> > > >
> > > > I would like to introduce second proposal.
> > > >
> > > > 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
> > > > - {db}/_all_docs
> > > > - {db}/_all_docs/queries
> > > > - _dbs_info
> > > > - {db}/_design/{ddoc}/_view/{view}
> > > > - {db}/_design/{ddoc}/_view/{view}/queries
> > > > 2) Add following additional fields into response:
> > > > ```
> > > > "first": {
> > > > "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
> > > > },
> > > > "previous": {
> > > > "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
> > > > },
> > > > "next": {
> > > > "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
> > > > },
> > > > ```
> > > > 3) Implement per-endpoint configurable max limits
> > > > ```
> > > > [request_limits]
> > > > _all_docs = 5000
> > > > _all_docs/queries = 5000
> > > > _all_dbs = 5000
> > > > _dbs_info = 5000
> > > > _view = 2500
> > > > _view/queries = 2500
> > > > _find = 2500
> > > > ```
> > > > 4) Implement following semantics:
> > > > - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
> > > > - don't use delayed responses when `bookmark` field is provided
> > > > - don't use delayed responses when `limit` query key is specified and when it is below the max limit
> > > > - return 400 when limit query key is specified and it is greater than the max limit
> > > > - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
> > > > - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
> > > >
> > > > Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
> > > >
> > > > Questions:
> > > > - `bookmark` vs `token`?
> > > > - should we prohibit setting other fields when bookmark is set?
> > > > - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
> > > >
> > > > Best regards,
> > > > iilyak
> > > >
> > > > On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
> > > > > Hello everyone,
> > > > >
> > > > > Based on the discussions on the thread I would like to propose a number of first steps:
> > > > > 1) introduce new endpoints
> > > > > - {db}/_all_docs/page
> > > > > - {db}/_all_docs/queries/page
> > > > > - _all_dbs/page
> > > > > - _dbs_info/page
> > > > > - {db}/_design/{ddoc}/_view/{view}/page
> > > > > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > > > - {db}/_find/page
> > > > >
> > > > > These new endpoints would act as follows:
> > > > > - don't use delayed responses
> > > > > - return object with following structure
> > > > > ```
> > > > > {
> > > > > "total": Total,
> > > > > "bookmark": base64 encoded opaque value,
> > > > > "completed": true | false,
> > > > > "update_seq": when available,
> > > > > "page": current page number,
> > > > > "items": [
> > > > > ]
> > > > > }
> > > > > ```
> > > > > - the bookmark would include following data (base64 or protobuff???):
> > > > > - direction
> > > > > - page
> > > > > - descending
> > > > > - endkey
> > > > > - endkey_docid
> > > > > - inclusive_end
> > > > > - startkey
> > > > > - startkey_docid
> > > > > - last_key
> > > > > - update_seq
> > > > > - timestamp
> > > > > ```
> > > > >
> > > > > 2) Implement per-endpoint configurable max limits
> > > > > ```
> > > > > _all_docs = 5000
> > > > > _all_docs/queries = 5000
> > > > > _all_dbs = 5000
> > > > > _dbs_info = 5000
> > > > > _view = 2500
> > > > > _view/queries = 2500
> > > > > _find = 2500
> > > > > ```
> > > > >
> > > > > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> > > > >
> > > > > Best regards,
> > > > > iilyak
> > > > >
> > > > > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > > > > Hello everyone,
> > > > > >
> > > > > > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> > > > > >
> > > > > > By "streaming APIs" I mean APIs which stream data in row as it gets
> > > > > > read from the database. These are the endpoints I was thinking of:
> > > > > >
> > > > > > _all_docs, _all_dbs, _dbs_info and query results
> > > > > >
> > > > > > I want to focus on what happens when FoundationDB transactions
> > > > > > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > > > > feeds, will crash or freeze. The reason is because the
> > > > > > transaction_too_old error at the end of 5 seconds is retry-able by
> > > > > > default, so the request handlers run again and end up shoving the
> > > > > > whole request down the socket again, headers and all, which is
> > > > > > obviously broken and not what we want.
> > > > > >
> > > > > > There are few alternatives discussed in couchdb-dev channel. I'll
> > > > > > present some behaviors but feel free to add more. Some ideas might
> > > > > > have been discounted on the IRC discussion already but I'll present
> > > > > > them anyway in case is sparks further conversation:
> > > > > >
> > > > > > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > > > > streaming the data from the next key after last emitted in the
> > > > > > previous transaction. Document the API behavior change that it may
> > > > > > present a view of the data is never a point-in-time[4] snapshot of the
> > > > > > DB.
> > > > > >
> > > > > > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > > > > don't have to change to continue using these CouchDB 4.0 endpoints
> > > > > > - This is the easiest to implement since it would re-use the
> > > > > > implementation for _changes feed (an extra option passed to the fold
> > > > > > function).
> > > > > > - Breaks API behavior if users relied on having a point-in-time[4]
> > > > > > snapshot view of the data.
> > > > > >
> > > > > > B) Simply end the stream. Let the users pass a `?transaction=true`
> > > > > > param which indicates they are aware the stream may end early and so
> > > > > > would have to paginate from the last emitted key with a skip=1. This
> > > > > > will keep the request bodies the same as current CouchDB. However, if
> > > > > > the users got all the data one request, they will end up wasting
> > > > > > another request to see if there is more data available. If they didn't
> > > > > > get any data they might have a too large of a skip value (see [2]) so
> > > > > > would have to guess different values for start/end keys. Or impose max
> > > > > > limit for the `skip` parameter.
> > > > > >
> > > > > > C) End the stream and add a final metadata row like a "transaction":
> > > > > > "timeout" at the end. That will let the user know to keep paginating
> > > > > > from the last key onward. This won't work for `_all_dbs` and
> > > > > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > > > > feeds and only use this for views and and _all_docs? If we like this
> > > > > > choice, let's think what happens for those as I couldn't come up with
> > > > > > anything decent there.
> > > > > >
> > > > > > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > > > > "key" of where the iteration stopped and the current "skip" and
> > > > > > "limit" params, which would keep decreasing. Then user would pass
> > > > > > those in "start_key=..." in the next request along with the limit and
> > > > > > skip params. So something like "continuation":{"skip":599, "limit":5,
> > > > > > "key":"..."}. This has the same issue with array results for
> > > > > > `_all_dbs` and `_dbs_info`[3].
> > > > > >
> > > > > > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > > > > there such that response time is likely to fit in one transaction.
> > > > > > This could be tricky as different runtime environments will have
> > > > > > different characteristics. Also, if the timeout happens there isn't a
> > > > > > a nice way to send an HTTP error since we already sent the 200
> > > > > > response. The downside is that this might break how some users use the
> > > > > > API, if say the are using large skips and limits already. Perhaps here
> > > > > > we do both B and D, such that if users want transactional behavior,
> > > > > > they specify that `transaction=true` param and only then we enforce
> > > > > > low limit and skip maximums.
> > > > > >
> > > > > > F) At least for `_all_docs` it seems providing a point-in-time
> > > > > > snapshot view doesn't necessarily need to be tied to transaction
> > > > > > boundaries. We could check the update sequence of the database at the
> > > > > > start of the next transaction and if it hasn't changed we can continue
> > > > > > emitting a consistent view. This can apply to C and D and would just
> > > > > > determine when the stream ends. If there are no writes happening to
> > > > > > the db, this could potential streams all the data just like option A
> > > > > > would do. Not entirely sure if this would work for views.
> > > > > >
> > > > > > So what do we think? I can see different combinations of options here,
> > > > > > maybe even different for each API point. For example `_all_dbs`,
> > > > > > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > > > > have parameters to do F, etc.
> > > > > >
> > > > > > Cheers,
> > > > > > -Nick
> > > > > >
> > > > > > Some footnotes:
> > > > > >
> > > > > > [1] _changes feeds is the only one that works currently. It behaves as
> > > > > > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > > > > > That is, we continue streaming the data by resetting the transaction
> > > > > > object and restarting from the last emitted key (db sequence in this
> > > > > > case). However, because the transaction restarts if a document is
> > > > > > updated while the streaming take place, it may appear in the _changes
> > > > > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > > > > have to document it, since previously we presented this point-in-time
> > > > > > snapshot of the database from when we started streaming.
> > > > > >
> > > > > > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > > > > currently support efficient offsets for key selectors
> > > > > > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > > > > > we implemented skip by iterating over the data. This means that a skip
> > > > > > of say 100000 could keep timing out the transaction without yielding
> > > > > > any data.
> > > > > >
> > > > > > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > > > > obvious place to insert a last metadata row.
> > > > > >
> > > > > > [4] For example they have a constraint that documents "a" and "z"
> > > > > > cannot both be in the database at the same time. But when iterating
> > > > > > it's possible that "a" was there at the start. Then by the end, "a"
> > > > > > was removed and "z" added, so both "a" and "z" would appear in the
> > > > > > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > > > > "relaxed" constrains:
> > > > > > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Nick Vatamaniuc <va...@gmail.com>.
Hi Ilya,
Maybe we should set a hard limit on the maximum doc ids size, 2-4KB?
We have a config setting to do it already. And we also have hard
limit of 10KB for FDB keys. Due to a limitation in Erlang http header
parser, used through mochiweb, 8KB is (was?) the limit based on the
default socket receive buffer size (see [1] for gory details).
I was going to suggest 4KB initially but say users would want to
specify a start_key and end_key + the dbname, so maybe 3KB or 2KB is a
better option? It will be an incompatible change so that's something
balance against.
Regarding setting server_name in the config, you're referring to the
specific limitation of 8KB from the mochweb header parsing, that
shouldn't include the domain name, just individual request + header
lines. It would look like "GET /dbname/_all_docs?bookmark...".
Also was curious, in the latest proposal what will be included in the
bookmark? Proposal said "The bookmark would include information needed
to ensure proper pagination without the need to repeat initial
parameters of the request".
[1] https://github.com/apache/couchdb/commit/d23025ebd7176f6c307ddf49902cf20b33bd55c4
On Thu, Apr 30, 2020 at 10:23 AM Ilya Khlopotov <ii...@apache.org> wrote:
>
> There is a problem with representing `next`/`previous`/`first` as path. With 5kB sized doc keys and we could exceed max URL length (8192 bytes). This means we would have to support POST. The question is how to handle the case when the URL is greater than 8192. The problem is CouchDB don't knows it's DNS name so we don't know the safe value to compare with.
>
> Options are:
> 1) always use POST for pagination
> 2) add server_name to config and return error when bookmark length exceeds dynamically calculated threshold. The threshold would account for db name length, server_name length, port and scheme length.
> - what error to return?
>
> I think option number 2 is too subtle to implement.
>
> The downside of option 1 is it is a bit harder to use from the browser or curl.
>
> On 2020/04/29 17:27:59, Ilya Khlopotov <ii...@apache.org> wrote:
> > I think I addressed all comments and created an RFC https://github.com/apache/couchdb-documentation/pull/530
> >
> > On 2020/04/28 11:56:15, Ilya Khlopotov <ii...@apache.org> wrote:
> > > Hello,
> > >
> > > I would like to introduce second proposal.
> > >
> > > 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
> > > - {db}/_all_docs
> > > - {db}/_all_docs/queries
> > > - _dbs_info
> > > - {db}/_design/{ddoc}/_view/{view}
> > > - {db}/_design/{ddoc}/_view/{view}/queries
> > > 2) Add following additional fields into response:
> > > ```
> > > "first": {
> > > "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
> > > },
> > > "previous": {
> > > "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
> > > },
> > > "next": {
> > > "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
> > > },
> > > ```
> > > 3) Implement per-endpoint configurable max limits
> > > ```
> > > [request_limits]
> > > _all_docs = 5000
> > > _all_docs/queries = 5000
> > > _all_dbs = 5000
> > > _dbs_info = 5000
> > > _view = 2500
> > > _view/queries = 2500
> > > _find = 2500
> > > ```
> > > 4) Implement following semantics:
> > > - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
> > > - don't use delayed responses when `bookmark` field is provided
> > > - don't use delayed responses when `limit` query key is specified and when it is below the max limit
> > > - return 400 when limit query key is specified and it is greater than the max limit
> > > - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
> > > - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
> > >
> > > Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
> > >
> > > Questions:
> > > - `bookmark` vs `token`?
> > > - should we prohibit setting other fields when bookmark is set?
> > > - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
> > >
> > > Best regards,
> > > iilyak
> > >
> > > On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
> > > > Hello everyone,
> > > >
> > > > Based on the discussions on the thread I would like to propose a number of first steps:
> > > > 1) introduce new endpoints
> > > > - {db}/_all_docs/page
> > > > - {db}/_all_docs/queries/page
> > > > - _all_dbs/page
> > > > - _dbs_info/page
> > > > - {db}/_design/{ddoc}/_view/{view}/page
> > > > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > > - {db}/_find/page
> > > >
> > > > These new endpoints would act as follows:
> > > > - don't use delayed responses
> > > > - return object with following structure
> > > > ```
> > > > {
> > > > "total": Total,
> > > > "bookmark": base64 encoded opaque value,
> > > > "completed": true | false,
> > > > "update_seq": when available,
> > > > "page": current page number,
> > > > "items": [
> > > > ]
> > > > }
> > > > ```
> > > > - the bookmark would include following data (base64 or protobuff???):
> > > > - direction
> > > > - page
> > > > - descending
> > > > - endkey
> > > > - endkey_docid
> > > > - inclusive_end
> > > > - startkey
> > > > - startkey_docid
> > > > - last_key
> > > > - update_seq
> > > > - timestamp
> > > > ```
> > > >
> > > > 2) Implement per-endpoint configurable max limits
> > > > ```
> > > > _all_docs = 5000
> > > > _all_docs/queries = 5000
> > > > _all_dbs = 5000
> > > > _dbs_info = 5000
> > > > _view = 2500
> > > > _view/queries = 2500
> > > > _find = 2500
> > > > ```
> > > >
> > > > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> > > >
> > > > Best regards,
> > > > iilyak
> > > >
> > > > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > > > Hello everyone,
> > > > >
> > > > > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> > > > >
> > > > > By "streaming APIs" I mean APIs which stream data in row as it gets
> > > > > read from the database. These are the endpoints I was thinking of:
> > > > >
> > > > > _all_docs, _all_dbs, _dbs_info and query results
> > > > >
> > > > > I want to focus on what happens when FoundationDB transactions
> > > > > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > > > feeds, will crash or freeze. The reason is because the
> > > > > transaction_too_old error at the end of 5 seconds is retry-able by
> > > > > default, so the request handlers run again and end up shoving the
> > > > > whole request down the socket again, headers and all, which is
> > > > > obviously broken and not what we want.
> > > > >
> > > > > There are few alternatives discussed in couchdb-dev channel. I'll
> > > > > present some behaviors but feel free to add more. Some ideas might
> > > > > have been discounted on the IRC discussion already but I'll present
> > > > > them anyway in case is sparks further conversation:
> > > > >
> > > > > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > > > streaming the data from the next key after last emitted in the
> > > > > previous transaction. Document the API behavior change that it may
> > > > > present a view of the data is never a point-in-time[4] snapshot of the
> > > > > DB.
> > > > >
> > > > > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > > > don't have to change to continue using these CouchDB 4.0 endpoints
> > > > > - This is the easiest to implement since it would re-use the
> > > > > implementation for _changes feed (an extra option passed to the fold
> > > > > function).
> > > > > - Breaks API behavior if users relied on having a point-in-time[4]
> > > > > snapshot view of the data.
> > > > >
> > > > > B) Simply end the stream. Let the users pass a `?transaction=true`
> > > > > param which indicates they are aware the stream may end early and so
> > > > > would have to paginate from the last emitted key with a skip=1. This
> > > > > will keep the request bodies the same as current CouchDB. However, if
> > > > > the users got all the data one request, they will end up wasting
> > > > > another request to see if there is more data available. If they didn't
> > > > > get any data they might have a too large of a skip value (see [2]) so
> > > > > would have to guess different values for start/end keys. Or impose max
> > > > > limit for the `skip` parameter.
> > > > >
> > > > > C) End the stream and add a final metadata row like a "transaction":
> > > > > "timeout" at the end. That will let the user know to keep paginating
> > > > > from the last key onward. This won't work for `_all_dbs` and
> > > > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > > > feeds and only use this for views and and _all_docs? If we like this
> > > > > choice, let's think what happens for those as I couldn't come up with
> > > > > anything decent there.
> > > > >
> > > > > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > > > "key" of where the iteration stopped and the current "skip" and
> > > > > "limit" params, which would keep decreasing. Then user would pass
> > > > > those in "start_key=..." in the next request along with the limit and
> > > > > skip params. So something like "continuation":{"skip":599, "limit":5,
> > > > > "key":"..."}. This has the same issue with array results for
> > > > > `_all_dbs` and `_dbs_info`[3].
> > > > >
> > > > > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > > > there such that response time is likely to fit in one transaction.
> > > > > This could be tricky as different runtime environments will have
> > > > > different characteristics. Also, if the timeout happens there isn't a
> > > > > a nice way to send an HTTP error since we already sent the 200
> > > > > response. The downside is that this might break how some users use the
> > > > > API, if say the are using large skips and limits already. Perhaps here
> > > > > we do both B and D, such that if users want transactional behavior,
> > > > > they specify that `transaction=true` param and only then we enforce
> > > > > low limit and skip maximums.
> > > > >
> > > > > F) At least for `_all_docs` it seems providing a point-in-time
> > > > > snapshot view doesn't necessarily need to be tied to transaction
> > > > > boundaries. We could check the update sequence of the database at the
> > > > > start of the next transaction and if it hasn't changed we can continue
> > > > > emitting a consistent view. This can apply to C and D and would just
> > > > > determine when the stream ends. If there are no writes happening to
> > > > > the db, this could potential streams all the data just like option A
> > > > > would do. Not entirely sure if this would work for views.
> > > > >
> > > > > So what do we think? I can see different combinations of options here,
> > > > > maybe even different for each API point. For example `_all_dbs`,
> > > > > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > > > have parameters to do F, etc.
> > > > >
> > > > > Cheers,
> > > > > -Nick
> > > > >
> > > > > Some footnotes:
> > > > >
> > > > > [1] _changes feeds is the only one that works currently. It behaves as
> > > > > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > > > > That is, we continue streaming the data by resetting the transaction
> > > > > object and restarting from the last emitted key (db sequence in this
> > > > > case). However, because the transaction restarts if a document is
> > > > > updated while the streaming take place, it may appear in the _changes
> > > > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > > > have to document it, since previously we presented this point-in-time
> > > > > snapshot of the database from when we started streaming.
> > > > >
> > > > > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > > > currently support efficient offsets for key selectors
> > > > > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > > > > we implemented skip by iterating over the data. This means that a skip
> > > > > of say 100000 could keep timing out the transaction without yielding
> > > > > any data.
> > > > >
> > > > > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > > > obvious place to insert a last metadata row.
> > > > >
> > > > > [4] For example they have a constraint that documents "a" and "z"
> > > > > cannot both be in the database at the same time. But when iterating
> > > > > it's possible that "a" was there at the start. Then by the end, "a"
> > > > > was removed and "z" added, so both "a" and "z" would appear in the
> > > > > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > > > "relaxed" constrains:
> > > > > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
There is a problem with representing `next`/`previous`/`first` as path. With 5kB sized doc keys and we could exceed max URL length (8192 bytes). This means we would have to support POST. The question is how to handle the case when the URL is greater than 8192. The problem is CouchDB don't knows it's DNS name so we don't know the safe value to compare with.
Options are:
1) always use POST for pagination
2) add server_name to config and return error when bookmark length exceeds dynamically calculated threshold. The threshold would account for db name length, server_name length, port and scheme length.
- what error to return?
I think option number 2 is too subtle to implement.
The downside of option 1 is it is a bit harder to use from the browser or curl.
On 2020/04/29 17:27:59, Ilya Khlopotov <ii...@apache.org> wrote:
> I think I addressed all comments and created an RFC https://github.com/apache/couchdb-documentation/pull/530
>
> On 2020/04/28 11:56:15, Ilya Khlopotov <ii...@apache.org> wrote:
> > Hello,
> >
> > I would like to introduce second proposal.
> >
> > 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
> > - {db}/_all_docs
> > - {db}/_all_docs/queries
> > - _dbs_info
> > - {db}/_design/{ddoc}/_view/{view}
> > - {db}/_design/{ddoc}/_view/{view}/queries
> > 2) Add following additional fields into response:
> > ```
> > "first": {
> > "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
> > },
> > "previous": {
> > "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
> > },
> > "next": {
> > "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
> > },
> > ```
> > 3) Implement per-endpoint configurable max limits
> > ```
> > [request_limits]
> > _all_docs = 5000
> > _all_docs/queries = 5000
> > _all_dbs = 5000
> > _dbs_info = 5000
> > _view = 2500
> > _view/queries = 2500
> > _find = 2500
> > ```
> > 4) Implement following semantics:
> > - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
> > - don't use delayed responses when `bookmark` field is provided
> > - don't use delayed responses when `limit` query key is specified and when it is below the max limit
> > - return 400 when limit query key is specified and it is greater than the max limit
> > - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
> > - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
> >
> > Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
> >
> > Questions:
> > - `bookmark` vs `token`?
> > - should we prohibit setting other fields when bookmark is set?
> > - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
> >
> > Best regards,
> > iilyak
> >
> > On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
> > > Hello everyone,
> > >
> > > Based on the discussions on the thread I would like to propose a number of first steps:
> > > 1) introduce new endpoints
> > > - {db}/_all_docs/page
> > > - {db}/_all_docs/queries/page
> > > - _all_dbs/page
> > > - _dbs_info/page
> > > - {db}/_design/{ddoc}/_view/{view}/page
> > > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > - {db}/_find/page
> > >
> > > These new endpoints would act as follows:
> > > - don't use delayed responses
> > > - return object with following structure
> > > ```
> > > {
> > > "total": Total,
> > > "bookmark": base64 encoded opaque value,
> > > "completed": true | false,
> > > "update_seq": when available,
> > > "page": current page number,
> > > "items": [
> > > ]
> > > }
> > > ```
> > > - the bookmark would include following data (base64 or protobuff???):
> > > - direction
> > > - page
> > > - descending
> > > - endkey
> > > - endkey_docid
> > > - inclusive_end
> > > - startkey
> > > - startkey_docid
> > > - last_key
> > > - update_seq
> > > - timestamp
> > > ```
> > >
> > > 2) Implement per-endpoint configurable max limits
> > > ```
> > > _all_docs = 5000
> > > _all_docs/queries = 5000
> > > _all_dbs = 5000
> > > _dbs_info = 5000
> > > _view = 2500
> > > _view/queries = 2500
> > > _find = 2500
> > > ```
> > >
> > > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> > >
> > > Best regards,
> > > iilyak
> > >
> > > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > > Hello everyone,
> > > >
> > > > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> > > >
> > > > By "streaming APIs" I mean APIs which stream data in row as it gets
> > > > read from the database. These are the endpoints I was thinking of:
> > > >
> > > > _all_docs, _all_dbs, _dbs_info and query results
> > > >
> > > > I want to focus on what happens when FoundationDB transactions
> > > > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > > feeds, will crash or freeze. The reason is because the
> > > > transaction_too_old error at the end of 5 seconds is retry-able by
> > > > default, so the request handlers run again and end up shoving the
> > > > whole request down the socket again, headers and all, which is
> > > > obviously broken and not what we want.
> > > >
> > > > There are few alternatives discussed in couchdb-dev channel. I'll
> > > > present some behaviors but feel free to add more. Some ideas might
> > > > have been discounted on the IRC discussion already but I'll present
> > > > them anyway in case is sparks further conversation:
> > > >
> > > > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > > streaming the data from the next key after last emitted in the
> > > > previous transaction. Document the API behavior change that it may
> > > > present a view of the data is never a point-in-time[4] snapshot of the
> > > > DB.
> > > >
> > > > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > > don't have to change to continue using these CouchDB 4.0 endpoints
> > > > - This is the easiest to implement since it would re-use the
> > > > implementation for _changes feed (an extra option passed to the fold
> > > > function).
> > > > - Breaks API behavior if users relied on having a point-in-time[4]
> > > > snapshot view of the data.
> > > >
> > > > B) Simply end the stream. Let the users pass a `?transaction=true`
> > > > param which indicates they are aware the stream may end early and so
> > > > would have to paginate from the last emitted key with a skip=1. This
> > > > will keep the request bodies the same as current CouchDB. However, if
> > > > the users got all the data one request, they will end up wasting
> > > > another request to see if there is more data available. If they didn't
> > > > get any data they might have a too large of a skip value (see [2]) so
> > > > would have to guess different values for start/end keys. Or impose max
> > > > limit for the `skip` parameter.
> > > >
> > > > C) End the stream and add a final metadata row like a "transaction":
> > > > "timeout" at the end. That will let the user know to keep paginating
> > > > from the last key onward. This won't work for `_all_dbs` and
> > > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > > feeds and only use this for views and and _all_docs? If we like this
> > > > choice, let's think what happens for those as I couldn't come up with
> > > > anything decent there.
> > > >
> > > > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > > "key" of where the iteration stopped and the current "skip" and
> > > > "limit" params, which would keep decreasing. Then user would pass
> > > > those in "start_key=..." in the next request along with the limit and
> > > > skip params. So something like "continuation":{"skip":599, "limit":5,
> > > > "key":"..."}. This has the same issue with array results for
> > > > `_all_dbs` and `_dbs_info`[3].
> > > >
> > > > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > > there such that response time is likely to fit in one transaction.
> > > > This could be tricky as different runtime environments will have
> > > > different characteristics. Also, if the timeout happens there isn't a
> > > > a nice way to send an HTTP error since we already sent the 200
> > > > response. The downside is that this might break how some users use the
> > > > API, if say the are using large skips and limits already. Perhaps here
> > > > we do both B and D, such that if users want transactional behavior,
> > > > they specify that `transaction=true` param and only then we enforce
> > > > low limit and skip maximums.
> > > >
> > > > F) At least for `_all_docs` it seems providing a point-in-time
> > > > snapshot view doesn't necessarily need to be tied to transaction
> > > > boundaries. We could check the update sequence of the database at the
> > > > start of the next transaction and if it hasn't changed we can continue
> > > > emitting a consistent view. This can apply to C and D and would just
> > > > determine when the stream ends. If there are no writes happening to
> > > > the db, this could potential streams all the data just like option A
> > > > would do. Not entirely sure if this would work for views.
> > > >
> > > > So what do we think? I can see different combinations of options here,
> > > > maybe even different for each API point. For example `_all_dbs`,
> > > > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > > have parameters to do F, etc.
> > > >
> > > > Cheers,
> > > > -Nick
> > > >
> > > > Some footnotes:
> > > >
> > > > [1] _changes feeds is the only one that works currently. It behaves as
> > > > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > > > That is, we continue streaming the data by resetting the transaction
> > > > object and restarting from the last emitted key (db sequence in this
> > > > case). However, because the transaction restarts if a document is
> > > > updated while the streaming take place, it may appear in the _changes
> > > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > > have to document it, since previously we presented this point-in-time
> > > > snapshot of the database from when we started streaming.
> > > >
> > > > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > > currently support efficient offsets for key selectors
> > > > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > > > we implemented skip by iterating over the data. This means that a skip
> > > > of say 100000 could keep timing out the transaction without yielding
> > > > any data.
> > > >
> > > > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > > obvious place to insert a last metadata row.
> > > >
> > > > [4] For example they have a constraint that documents "a" and "z"
> > > > cannot both be in the database at the same time. But when iterating
> > > > it's possible that "a" was there at the start. Then by the end, "a"
> > > > was removed and "z" added, so both "a" and "z" would appear in the
> > > > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > > "relaxed" constrains:
> > > > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > > >
> > >
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
I think I addressed all comments and created an RFC https://github.com/apache/couchdb-documentation/pull/530
On 2020/04/28 11:56:15, Ilya Khlopotov <ii...@apache.org> wrote:
> Hello,
>
> I would like to introduce second proposal.
>
> 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
> - {db}/_all_docs
> - {db}/_all_docs/queries
> - _dbs_info
> - {db}/_design/{ddoc}/_view/{view}
> - {db}/_design/{ddoc}/_view/{view}/queries
> 2) Add following additional fields into response:
> ```
> "first": {
> "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
> },
> "previous": {
> "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
> },
> "next": {
> "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
> },
> ```
> 3) Implement per-endpoint configurable max limits
> ```
> [request_limits]
> _all_docs = 5000
> _all_docs/queries = 5000
> _all_dbs = 5000
> _dbs_info = 5000
> _view = 2500
> _view/queries = 2500
> _find = 2500
> ```
> 4) Implement following semantics:
> - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
> - don't use delayed responses when `bookmark` field is provided
> - don't use delayed responses when `limit` query key is specified and when it is below the max limit
> - return 400 when limit query key is specified and it is greater than the max limit
> - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
> - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
>
> Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
>
> Questions:
> - `bookmark` vs `token`?
> - should we prohibit setting other fields when bookmark is set?
> - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
>
> Best regards,
> iilyak
>
> On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
> > Hello everyone,
> >
> > Based on the discussions on the thread I would like to propose a number of first steps:
> > 1) introduce new endpoints
> > - {db}/_all_docs/page
> > - {db}/_all_docs/queries/page
> > - _all_dbs/page
> > - _dbs_info/page
> > - {db}/_design/{ddoc}/_view/{view}/page
> > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > - {db}/_find/page
> >
> > These new endpoints would act as follows:
> > - don't use delayed responses
> > - return object with following structure
> > ```
> > {
> > "total": Total,
> > "bookmark": base64 encoded opaque value,
> > "completed": true | false,
> > "update_seq": when available,
> > "page": current page number,
> > "items": [
> > ]
> > }
> > ```
> > - the bookmark would include following data (base64 or protobuff???):
> > - direction
> > - page
> > - descending
> > - endkey
> > - endkey_docid
> > - inclusive_end
> > - startkey
> > - startkey_docid
> > - last_key
> > - update_seq
> > - timestamp
> > ```
> >
> > 2) Implement per-endpoint configurable max limits
> > ```
> > _all_docs = 5000
> > _all_docs/queries = 5000
> > _all_dbs = 5000
> > _dbs_info = 5000
> > _view = 2500
> > _view/queries = 2500
> > _find = 2500
> > ```
> >
> > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> >
> > Best regards,
> > iilyak
> >
> > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > Hello everyone,
> > >
> > > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> > >
> > > By "streaming APIs" I mean APIs which stream data in row as it gets
> > > read from the database. These are the endpoints I was thinking of:
> > >
> > > _all_docs, _all_dbs, _dbs_info and query results
> > >
> > > I want to focus on what happens when FoundationDB transactions
> > > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > feeds, will crash or freeze. The reason is because the
> > > transaction_too_old error at the end of 5 seconds is retry-able by
> > > default, so the request handlers run again and end up shoving the
> > > whole request down the socket again, headers and all, which is
> > > obviously broken and not what we want.
> > >
> > > There are few alternatives discussed in couchdb-dev channel. I'll
> > > present some behaviors but feel free to add more. Some ideas might
> > > have been discounted on the IRC discussion already but I'll present
> > > them anyway in case is sparks further conversation:
> > >
> > > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > streaming the data from the next key after last emitted in the
> > > previous transaction. Document the API behavior change that it may
> > > present a view of the data is never a point-in-time[4] snapshot of the
> > > DB.
> > >
> > > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > don't have to change to continue using these CouchDB 4.0 endpoints
> > > - This is the easiest to implement since it would re-use the
> > > implementation for _changes feed (an extra option passed to the fold
> > > function).
> > > - Breaks API behavior if users relied on having a point-in-time[4]
> > > snapshot view of the data.
> > >
> > > B) Simply end the stream. Let the users pass a `?transaction=true`
> > > param which indicates they are aware the stream may end early and so
> > > would have to paginate from the last emitted key with a skip=1. This
> > > will keep the request bodies the same as current CouchDB. However, if
> > > the users got all the data one request, they will end up wasting
> > > another request to see if there is more data available. If they didn't
> > > get any data they might have a too large of a skip value (see [2]) so
> > > would have to guess different values for start/end keys. Or impose max
> > > limit for the `skip` parameter.
> > >
> > > C) End the stream and add a final metadata row like a "transaction":
> > > "timeout" at the end. That will let the user know to keep paginating
> > > from the last key onward. This won't work for `_all_dbs` and
> > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > feeds and only use this for views and and _all_docs? If we like this
> > > choice, let's think what happens for those as I couldn't come up with
> > > anything decent there.
> > >
> > > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > "key" of where the iteration stopped and the current "skip" and
> > > "limit" params, which would keep decreasing. Then user would pass
> > > those in "start_key=..." in the next request along with the limit and
> > > skip params. So something like "continuation":{"skip":599, "limit":5,
> > > "key":"..."}. This has the same issue with array results for
> > > `_all_dbs` and `_dbs_info`[3].
> > >
> > > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > there such that response time is likely to fit in one transaction.
> > > This could be tricky as different runtime environments will have
> > > different characteristics. Also, if the timeout happens there isn't a
> > > a nice way to send an HTTP error since we already sent the 200
> > > response. The downside is that this might break how some users use the
> > > API, if say the are using large skips and limits already. Perhaps here
> > > we do both B and D, such that if users want transactional behavior,
> > > they specify that `transaction=true` param and only then we enforce
> > > low limit and skip maximums.
> > >
> > > F) At least for `_all_docs` it seems providing a point-in-time
> > > snapshot view doesn't necessarily need to be tied to transaction
> > > boundaries. We could check the update sequence of the database at the
> > > start of the next transaction and if it hasn't changed we can continue
> > > emitting a consistent view. This can apply to C and D and would just
> > > determine when the stream ends. If there are no writes happening to
> > > the db, this could potential streams all the data just like option A
> > > would do. Not entirely sure if this would work for views.
> > >
> > > So what do we think? I can see different combinations of options here,
> > > maybe even different for each API point. For example `_all_dbs`,
> > > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > have parameters to do F, etc.
> > >
> > > Cheers,
> > > -Nick
> > >
> > > Some footnotes:
> > >
> > > [1] _changes feeds is the only one that works currently. It behaves as
> > > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > > That is, we continue streaming the data by resetting the transaction
> > > object and restarting from the last emitted key (db sequence in this
> > > case). However, because the transaction restarts if a document is
> > > updated while the streaming take place, it may appear in the _changes
> > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > have to document it, since previously we presented this point-in-time
> > > snapshot of the database from when we started streaming.
> > >
> > > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > currently support efficient offsets for key selectors
> > > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > > we implemented skip by iterating over the data. This means that a skip
> > > of say 100000 could keep timing out the transaction without yielding
> > > any data.
> > >
> > > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > obvious place to insert a last metadata row.
> > >
> > > [4] For example they have a constraint that documents "a" and "z"
> > > cannot both be in the database at the same time. But when iterating
> > > it's possible that "a" was there at the start. Then by the end, "a"
> > > was removed and "z" added, so both "a" and "z" would appear in the
> > > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > "relaxed" constrains:
> > > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > >
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Nick Vatamaniuc <va...@gmail.com>.
The `{"href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"}`
might be tricky if requests have to go through a few reverse proxies
before reaching CouchDB. CouchDB might not know its own "external"
domain so to speak. I have used X-Forwarded-For before for this exact
pattern before, and there is even an RFC for it now
https://tools.ietf.org/html/rfc7239, but it can be rather fragile.
On `bookmark` vs `token`, I like bookmark, cursor, iterator/iterate
better than token I think. Token us used in authentication /
authorization so it might get confusing.
On the `previous`/`next`/`first` wonder if `previous` and makes sense
if we already know the direction they are iterating toward based on
the initial request. Or do you propose that the bookmark doesn't
encode the direction anymore?
Thinking more about this and what's included in the bookmark, it
doesn't even need to include the limit. A user could change their
limit from request to request even when using a bookmark. If we don't
include the direction of iteration, we are down to just the start_key
that actually keeps changing in the bookmark and it would be kinda
silly to b64 encode the start key by itself in an opaque blob? So
going with that, we could just have a matching parameter like
inclusive_start=false (to match inclusive_end=) and let users pick off
the key from the result and use that as the start_key with
inclusive_start=false so they continue iterating from were they
stopped.
Also I think _dbs_info emits an array as well. We just recently added
a GET method for it to match _list_dbs.
On Tue, Apr 28, 2020 at 12:38 PM Paul Davis <pa...@gmail.com> wrote:
>
> Seems reasonable to me. I'd agree that setting query string parameters
> with a bookmark should be rejected. I was also going to suggest
> eliding the href member. In the examples I've seen those are usually
> structured as something like:
>
> "links": {
> "previous": "/path/and/qs=foo",
> "next": "/path/and/qs=bar"
> }
>
> On Tue, Apr 28, 2020 at 6:56 AM Ilya Khlopotov <ii...@apache.org> wrote:
> >
> > Hello,
> >
> > I would like to introduce second proposal.
> >
> > 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
> > - {db}/_all_docs
> > - {db}/_all_docs/queries
> > - _dbs_info
> > - {db}/_design/{ddoc}/_view/{view}
> > - {db}/_design/{ddoc}/_view/{view}/queries
> > 2) Add following additional fields into response:
> > ```
> > "first": {
> > "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
> > },
> > "previous": {
> > "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
> > },
> > "next": {
> > "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
> > },
> > ```
> > 3) Implement per-endpoint configurable max limits
> > ```
> > [request_limits]
> > _all_docs = 5000
> > _all_docs/queries = 5000
> > _all_dbs = 5000
> > _dbs_info = 5000
> > _view = 2500
> > _view/queries = 2500
> > _find = 2500
> > ```
> > 4) Implement following semantics:
> > - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
> > - don't use delayed responses when `bookmark` field is provided
> > - don't use delayed responses when `limit` query key is specified and when it is below the max limit
> > - return 400 when limit query key is specified and it is greater than the max limit
> > - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
> > - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
> >
> > Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
> >
> > Questions:
> > - `bookmark` vs `token`?
> > - should we prohibit setting other fields when bookmark is set?
> > - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
> >
> > Best regards,
> > iilyak
> >
> > On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
> > > Hello everyone,
> > >
> > > Based on the discussions on the thread I would like to propose a number of first steps:
> > > 1) introduce new endpoints
> > > - {db}/_all_docs/page
> > > - {db}/_all_docs/queries/page
> > > - _all_dbs/page
> > > - _dbs_info/page
> > > - {db}/_design/{ddoc}/_view/{view}/page
> > > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > - {db}/_find/page
> > >
> > > These new endpoints would act as follows:
> > > - don't use delayed responses
> > > - return object with following structure
> > > ```
> > > {
> > > "total": Total,
> > > "bookmark": base64 encoded opaque value,
> > > "completed": true | false,
> > > "update_seq": when available,
> > > "page": current page number,
> > > "items": [
> > > ]
> > > }
> > > ```
> > > - the bookmark would include following data (base64 or protobuff???):
> > > - direction
> > > - page
> > > - descending
> > > - endkey
> > > - endkey_docid
> > > - inclusive_end
> > > - startkey
> > > - startkey_docid
> > > - last_key
> > > - update_seq
> > > - timestamp
> > > ```
> > >
> > > 2) Implement per-endpoint configurable max limits
> > > ```
> > > _all_docs = 5000
> > > _all_docs/queries = 5000
> > > _all_dbs = 5000
> > > _dbs_info = 5000
> > > _view = 2500
> > > _view/queries = 2500
> > > _find = 2500
> > > ```
> > >
> > > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> > >
> > > Best regards,
> > > iilyak
> > >
> > > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > > Hello everyone,
> > > >
> > > > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> > > >
> > > > By "streaming APIs" I mean APIs which stream data in row as it gets
> > > > read from the database. These are the endpoints I was thinking of:
> > > >
> > > > _all_docs, _all_dbs, _dbs_info and query results
> > > >
> > > > I want to focus on what happens when FoundationDB transactions
> > > > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > > feeds, will crash or freeze. The reason is because the
> > > > transaction_too_old error at the end of 5 seconds is retry-able by
> > > > default, so the request handlers run again and end up shoving the
> > > > whole request down the socket again, headers and all, which is
> > > > obviously broken and not what we want.
> > > >
> > > > There are few alternatives discussed in couchdb-dev channel. I'll
> > > > present some behaviors but feel free to add more. Some ideas might
> > > > have been discounted on the IRC discussion already but I'll present
> > > > them anyway in case is sparks further conversation:
> > > >
> > > > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > > streaming the data from the next key after last emitted in the
> > > > previous transaction. Document the API behavior change that it may
> > > > present a view of the data is never a point-in-time[4] snapshot of the
> > > > DB.
> > > >
> > > > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > > don't have to change to continue using these CouchDB 4.0 endpoints
> > > > - This is the easiest to implement since it would re-use the
> > > > implementation for _changes feed (an extra option passed to the fold
> > > > function).
> > > > - Breaks API behavior if users relied on having a point-in-time[4]
> > > > snapshot view of the data.
> > > >
> > > > B) Simply end the stream. Let the users pass a `?transaction=true`
> > > > param which indicates they are aware the stream may end early and so
> > > > would have to paginate from the last emitted key with a skip=1. This
> > > > will keep the request bodies the same as current CouchDB. However, if
> > > > the users got all the data one request, they will end up wasting
> > > > another request to see if there is more data available. If they didn't
> > > > get any data they might have a too large of a skip value (see [2]) so
> > > > would have to guess different values for start/end keys. Or impose max
> > > > limit for the `skip` parameter.
> > > >
> > > > C) End the stream and add a final metadata row like a "transaction":
> > > > "timeout" at the end. That will let the user know to keep paginating
> > > > from the last key onward. This won't work for `_all_dbs` and
> > > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > > feeds and only use this for views and and _all_docs? If we like this
> > > > choice, let's think what happens for those as I couldn't come up with
> > > > anything decent there.
> > > >
> > > > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > > "key" of where the iteration stopped and the current "skip" and
> > > > "limit" params, which would keep decreasing. Then user would pass
> > > > those in "start_key=..." in the next request along with the limit and
> > > > skip params. So something like "continuation":{"skip":599, "limit":5,
> > > > "key":"..."}. This has the same issue with array results for
> > > > `_all_dbs` and `_dbs_info`[3].
> > > >
> > > > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > > there such that response time is likely to fit in one transaction.
> > > > This could be tricky as different runtime environments will have
> > > > different characteristics. Also, if the timeout happens there isn't a
> > > > a nice way to send an HTTP error since we already sent the 200
> > > > response. The downside is that this might break how some users use the
> > > > API, if say the are using large skips and limits already. Perhaps here
> > > > we do both B and D, such that if users want transactional behavior,
> > > > they specify that `transaction=true` param and only then we enforce
> > > > low limit and skip maximums.
> > > >
> > > > F) At least for `_all_docs` it seems providing a point-in-time
> > > > snapshot view doesn't necessarily need to be tied to transaction
> > > > boundaries. We could check the update sequence of the database at the
> > > > start of the next transaction and if it hasn't changed we can continue
> > > > emitting a consistent view. This can apply to C and D and would just
> > > > determine when the stream ends. If there are no writes happening to
> > > > the db, this could potential streams all the data just like option A
> > > > would do. Not entirely sure if this would work for views.
> > > >
> > > > So what do we think? I can see different combinations of options here,
> > > > maybe even different for each API point. For example `_all_dbs`,
> > > > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > > have parameters to do F, etc.
> > > >
> > > > Cheers,
> > > > -Nick
> > > >
> > > > Some footnotes:
> > > >
> > > > [1] _changes feeds is the only one that works currently. It behaves as
> > > > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > > > That is, we continue streaming the data by resetting the transaction
> > > > object and restarting from the last emitted key (db sequence in this
> > > > case). However, because the transaction restarts if a document is
> > > > updated while the streaming take place, it may appear in the _changes
> > > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > > have to document it, since previously we presented this point-in-time
> > > > snapshot of the database from when we started streaming.
> > > >
> > > > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > > currently support efficient offsets for key selectors
> > > > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > > > we implemented skip by iterating over the data. This means that a skip
> > > > of say 100000 could keep timing out the transaction without yielding
> > > > any data.
> > > >
> > > > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > > obvious place to insert a last metadata row.
> > > >
> > > > [4] For example they have a constraint that documents "a" and "z"
> > > > cannot both be in the database at the same time. But when iterating
> > > > it's possible that "a" was there at the start. Then by the end, "a"
> > > > was removed and "z" added, so both "a" and "z" would appear in the
> > > > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > > "relaxed" constrains:
> > > > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > > >
> > >
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Paul Davis <pa...@gmail.com>.
Seems reasonable to me. I'd agree that setting query string parameters
with a bookmark should be rejected. I was also going to suggest
eliding the href member. In the examples I've seen those are usually
structured as something like:
"links": {
"previous": "/path/and/qs=foo",
"next": "/path/and/qs=bar"
}
On Tue, Apr 28, 2020 at 6:56 AM Ilya Khlopotov <ii...@apache.org> wrote:
>
> Hello,
>
> I would like to introduce second proposal.
>
> 1) Add new optional query field called `bookmark` (or `token`) to following endpoints
> - {db}/_all_docs
> - {db}/_all_docs/queries
> - _dbs_info
> - {db}/_design/{ddoc}/_view/{view}
> - {db}/_design/{ddoc}/_view/{view}/queries
> 2) Add following additional fields into response:
> ```
> "first": {
> "href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
> },
> "previous": {
> "href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
> },
> "next": {
> "href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
> },
> ```
> 3) Implement per-endpoint configurable max limits
> ```
> [request_limits]
> _all_docs = 5000
> _all_docs/queries = 5000
> _all_dbs = 5000
> _dbs_info = 5000
> _view = 2500
> _view/queries = 2500
> _find = 2500
> ```
> 4) Implement following semantics:
> - The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
> - don't use delayed responses when `bookmark` field is provided
> - don't use delayed responses when `limit` query key is specified and when it is below the max limit
> - return 400 when limit query key is specified and it is greater than the max limit
> - return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
> - the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
>
> Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
>
> Questions:
> - `bookmark` vs `token`?
> - should we prohibit setting other fields when bookmark is set?
> - `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
>
> Best regards,
> iilyak
>
> On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
> > Hello everyone,
> >
> > Based on the discussions on the thread I would like to propose a number of first steps:
> > 1) introduce new endpoints
> > - {db}/_all_docs/page
> > - {db}/_all_docs/queries/page
> > - _all_dbs/page
> > - _dbs_info/page
> > - {db}/_design/{ddoc}/_view/{view}/page
> > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > - {db}/_find/page
> >
> > These new endpoints would act as follows:
> > - don't use delayed responses
> > - return object with following structure
> > ```
> > {
> > "total": Total,
> > "bookmark": base64 encoded opaque value,
> > "completed": true | false,
> > "update_seq": when available,
> > "page": current page number,
> > "items": [
> > ]
> > }
> > ```
> > - the bookmark would include following data (base64 or protobuff???):
> > - direction
> > - page
> > - descending
> > - endkey
> > - endkey_docid
> > - inclusive_end
> > - startkey
> > - startkey_docid
> > - last_key
> > - update_seq
> > - timestamp
> > ```
> >
> > 2) Implement per-endpoint configurable max limits
> > ```
> > _all_docs = 5000
> > _all_docs/queries = 5000
> > _all_dbs = 5000
> > _dbs_info = 5000
> > _view = 2500
> > _view/queries = 2500
> > _find = 2500
> > ```
> >
> > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> >
> > Best regards,
> > iilyak
> >
> > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > Hello everyone,
> > >
> > > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> > >
> > > By "streaming APIs" I mean APIs which stream data in row as it gets
> > > read from the database. These are the endpoints I was thinking of:
> > >
> > > _all_docs, _all_dbs, _dbs_info and query results
> > >
> > > I want to focus on what happens when FoundationDB transactions
> > > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > feeds, will crash or freeze. The reason is because the
> > > transaction_too_old error at the end of 5 seconds is retry-able by
> > > default, so the request handlers run again and end up shoving the
> > > whole request down the socket again, headers and all, which is
> > > obviously broken and not what we want.
> > >
> > > There are few alternatives discussed in couchdb-dev channel. I'll
> > > present some behaviors but feel free to add more. Some ideas might
> > > have been discounted on the IRC discussion already but I'll present
> > > them anyway in case is sparks further conversation:
> > >
> > > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > streaming the data from the next key after last emitted in the
> > > previous transaction. Document the API behavior change that it may
> > > present a view of the data is never a point-in-time[4] snapshot of the
> > > DB.
> > >
> > > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > don't have to change to continue using these CouchDB 4.0 endpoints
> > > - This is the easiest to implement since it would re-use the
> > > implementation for _changes feed (an extra option passed to the fold
> > > function).
> > > - Breaks API behavior if users relied on having a point-in-time[4]
> > > snapshot view of the data.
> > >
> > > B) Simply end the stream. Let the users pass a `?transaction=true`
> > > param which indicates they are aware the stream may end early and so
> > > would have to paginate from the last emitted key with a skip=1. This
> > > will keep the request bodies the same as current CouchDB. However, if
> > > the users got all the data one request, they will end up wasting
> > > another request to see if there is more data available. If they didn't
> > > get any data they might have a too large of a skip value (see [2]) so
> > > would have to guess different values for start/end keys. Or impose max
> > > limit for the `skip` parameter.
> > >
> > > C) End the stream and add a final metadata row like a "transaction":
> > > "timeout" at the end. That will let the user know to keep paginating
> > > from the last key onward. This won't work for `_all_dbs` and
> > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > feeds and only use this for views and and _all_docs? If we like this
> > > choice, let's think what happens for those as I couldn't come up with
> > > anything decent there.
> > >
> > > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > "key" of where the iteration stopped and the current "skip" and
> > > "limit" params, which would keep decreasing. Then user would pass
> > > those in "start_key=..." in the next request along with the limit and
> > > skip params. So something like "continuation":{"skip":599, "limit":5,
> > > "key":"..."}. This has the same issue with array results for
> > > `_all_dbs` and `_dbs_info`[3].
> > >
> > > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > there such that response time is likely to fit in one transaction.
> > > This could be tricky as different runtime environments will have
> > > different characteristics. Also, if the timeout happens there isn't a
> > > a nice way to send an HTTP error since we already sent the 200
> > > response. The downside is that this might break how some users use the
> > > API, if say the are using large skips and limits already. Perhaps here
> > > we do both B and D, such that if users want transactional behavior,
> > > they specify that `transaction=true` param and only then we enforce
> > > low limit and skip maximums.
> > >
> > > F) At least for `_all_docs` it seems providing a point-in-time
> > > snapshot view doesn't necessarily need to be tied to transaction
> > > boundaries. We could check the update sequence of the database at the
> > > start of the next transaction and if it hasn't changed we can continue
> > > emitting a consistent view. This can apply to C and D and would just
> > > determine when the stream ends. If there are no writes happening to
> > > the db, this could potential streams all the data just like option A
> > > would do. Not entirely sure if this would work for views.
> > >
> > > So what do we think? I can see different combinations of options here,
> > > maybe even different for each API point. For example `_all_dbs`,
> > > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > have parameters to do F, etc.
> > >
> > > Cheers,
> > > -Nick
> > >
> > > Some footnotes:
> > >
> > > [1] _changes feeds is the only one that works currently. It behaves as
> > > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > > That is, we continue streaming the data by resetting the transaction
> > > object and restarting from the last emitted key (db sequence in this
> > > case). However, because the transaction restarts if a document is
> > > updated while the streaming take place, it may appear in the _changes
> > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > have to document it, since previously we presented this point-in-time
> > > snapshot of the database from when we started streaming.
> > >
> > > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > currently support efficient offsets for key selectors
> > > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > > we implemented skip by iterating over the data. This means that a skip
> > > of say 100000 could keep timing out the transaction without yielding
> > > any data.
> > >
> > > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > obvious place to insert a last metadata row.
> > >
> > > [4] For example they have a constraint that documents "a" and "z"
> > > cannot both be in the database at the same time. But when iterating
> > > it's possible that "a" was there at the start. Then by the end, "a"
> > > was removed and "z" added, so both "a" and "z" would appear in the
> > > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > "relaxed" constrains:
> > > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > >
> >
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
Hello,
I would like to introduce second proposal.
1) Add new optional query field called `bookmark` (or `token`) to following endpoints
- {db}/_all_docs
- {db}/_all_docs/queries
- _dbs_info
- {db}/_design/{ddoc}/_view/{view}
- {db}/_design/{ddoc}/_view/{view}/queries
2) Add following additional fields into response:
```
"first": {
"href": "https://myserver.com/myddb/_all_docs?limit=50&descending=true"
},
"previous": {
"href": "https://myserver.com/myddb/_all_docs?bookmark=983uiwfjkdsdf"
},
"next": {
"href": "https://myserver.com/myddb/_all_docs?bookmark=12343tyekf3"
},
```
3) Implement per-endpoint configurable max limits
```
[request_limits]
_all_docs = 5000
_all_docs/queries = 5000
_all_dbs = 5000
_dbs_info = 5000
_view = 2500
_view/queries = 2500
_find = 2500
```
4) Implement following semantics:
- The bookmark would be opaque token and would include information needed to ensure proper pagination without the need to repeat initial parameters of the request. In fact we might prohibit setting additional parameters when bookmark query field is specified.
- don't use delayed responses when `bookmark` field is provided
- don't use delayed responses when `limit` query key is specified and when it is below the max limit
- return 400 when limit query key is specified and it is greater than the max limit
- return 400 when we stream rows (in case when `limit` query key wasn't specified) and reach max limit
- the `previous`/`next`/`first` keys are optional and we omit them for the cases they don't make sense
Latter on we would introduce API versioning and deal with `{db}/_changes` and `_all_docs` endpoints.
Questions:
- `bookmark` vs `token`?
- should we prohibit setting other fields when bookmark is set?
- `previous`/`next`/`first` as href vs token value itself (i.e. `{"previous": "983uiwfjkdsdf", "next": "12343tyekf3", "first": "iekjhfwo034"}`)
Best regards,
iilyak
On 2020/04/22 20:18:57, Ilya Khlopotov <ii...@apache.org> wrote:
> Hello everyone,
>
> Based on the discussions on the thread I would like to propose a number of first steps:
> 1) introduce new endpoints
> - {db}/_all_docs/page
> - {db}/_all_docs/queries/page
> - _all_dbs/page
> - _dbs_info/page
> - {db}/_design/{ddoc}/_view/{view}/page
> - {db}/_design/{ddoc}/_view/{view}/queries/page
> - {db}/_find/page
>
> These new endpoints would act as follows:
> - don't use delayed responses
> - return object with following structure
> ```
> {
> "total": Total,
> "bookmark": base64 encoded opaque value,
> "completed": true | false,
> "update_seq": when available,
> "page": current page number,
> "items": [
> ]
> }
> ```
> - the bookmark would include following data (base64 or protobuff???):
> - direction
> - page
> - descending
> - endkey
> - endkey_docid
> - inclusive_end
> - startkey
> - startkey_docid
> - last_key
> - update_seq
> - timestamp
> ```
>
> 2) Implement per-endpoint configurable max limits
> ```
> _all_docs = 5000
> _all_docs/queries = 5000
> _all_dbs = 5000
> _dbs_info = 5000
> _view = 2500
> _view/queries = 2500
> _find = 2500
> ```
>
> Latter (after few years) CouchDB would deprecate and remove old endpoints.
>
> Best regards,
> iilyak
>
> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > Hello everyone,
> >
> > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> >
> > By "streaming APIs" I mean APIs which stream data in row as it gets
> > read from the database. These are the endpoints I was thinking of:
> >
> > _all_docs, _all_dbs, _dbs_info and query results
> >
> > I want to focus on what happens when FoundationDB transactions
> > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > feeds, will crash or freeze. The reason is because the
> > transaction_too_old error at the end of 5 seconds is retry-able by
> > default, so the request handlers run again and end up shoving the
> > whole request down the socket again, headers and all, which is
> > obviously broken and not what we want.
> >
> > There are few alternatives discussed in couchdb-dev channel. I'll
> > present some behaviors but feel free to add more. Some ideas might
> > have been discounted on the IRC discussion already but I'll present
> > them anyway in case is sparks further conversation:
> >
> > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > streaming the data from the next key after last emitted in the
> > previous transaction. Document the API behavior change that it may
> > present a view of the data is never a point-in-time[4] snapshot of the
> > DB.
> >
> > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > don't have to change to continue using these CouchDB 4.0 endpoints
> > - This is the easiest to implement since it would re-use the
> > implementation for _changes feed (an extra option passed to the fold
> > function).
> > - Breaks API behavior if users relied on having a point-in-time[4]
> > snapshot view of the data.
> >
> > B) Simply end the stream. Let the users pass a `?transaction=true`
> > param which indicates they are aware the stream may end early and so
> > would have to paginate from the last emitted key with a skip=1. This
> > will keep the request bodies the same as current CouchDB. However, if
> > the users got all the data one request, they will end up wasting
> > another request to see if there is more data available. If they didn't
> > get any data they might have a too large of a skip value (see [2]) so
> > would have to guess different values for start/end keys. Or impose max
> > limit for the `skip` parameter.
> >
> > C) End the stream and add a final metadata row like a "transaction":
> > "timeout" at the end. That will let the user know to keep paginating
> > from the last key onward. This won't work for `_all_dbs` and
> > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > feeds and only use this for views and and _all_docs? If we like this
> > choice, let's think what happens for those as I couldn't come up with
> > anything decent there.
> >
> > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > "key" of where the iteration stopped and the current "skip" and
> > "limit" params, which would keep decreasing. Then user would pass
> > those in "start_key=..." in the next request along with the limit and
> > skip params. So something like "continuation":{"skip":599, "limit":5,
> > "key":"..."}. This has the same issue with array results for
> > `_all_dbs` and `_dbs_info`[3].
> >
> > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > there such that response time is likely to fit in one transaction.
> > This could be tricky as different runtime environments will have
> > different characteristics. Also, if the timeout happens there isn't a
> > a nice way to send an HTTP error since we already sent the 200
> > response. The downside is that this might break how some users use the
> > API, if say the are using large skips and limits already. Perhaps here
> > we do both B and D, such that if users want transactional behavior,
> > they specify that `transaction=true` param and only then we enforce
> > low limit and skip maximums.
> >
> > F) At least for `_all_docs` it seems providing a point-in-time
> > snapshot view doesn't necessarily need to be tied to transaction
> > boundaries. We could check the update sequence of the database at the
> > start of the next transaction and if it hasn't changed we can continue
> > emitting a consistent view. This can apply to C and D and would just
> > determine when the stream ends. If there are no writes happening to
> > the db, this could potential streams all the data just like option A
> > would do. Not entirely sure if this would work for views.
> >
> > So what do we think? I can see different combinations of options here,
> > maybe even different for each API point. For example `_all_dbs`,
> > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > have parameters to do F, etc.
> >
> > Cheers,
> > -Nick
> >
> > Some footnotes:
> >
> > [1] _changes feeds is the only one that works currently. It behaves as
> > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > That is, we continue streaming the data by resetting the transaction
> > object and restarting from the last emitted key (db sequence in this
> > case). However, because the transaction restarts if a document is
> > updated while the streaming take place, it may appear in the _changes
> > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > have to document it, since previously we presented this point-in-time
> > snapshot of the database from when we started streaming.
> >
> > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > currently support efficient offsets for key selectors
> > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > we implemented skip by iterating over the data. This means that a skip
> > of say 100000 could keep timing out the transaction without yielding
> > any data.
> >
> > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > obvious place to insert a last metadata row.
> >
> > [4] For example they have a constraint that documents "a" and "z"
> > cannot both be in the database at the same time. But when iterating
> > it's possible that "a" was there at the start. Then by the end, "a"
> > was removed and "z" added, so both "a" and "z" would appear in the
> > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > "relaxed" constrains:
> > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Nick Vatamaniuc <va...@gmail.com>.
Hi Ilya,
Good idea. Let's continue the discussion about versioning there.
On Mon, Apr 27, 2020 at 12:51 PM Ilya Khlopotov <ii...@apache.org> wrote:
>
> Hi Nick,
>
> Thank you for extensive answer.
>
> > API versioning idea in principle sounds good, but can't think of a
> > clean way to do it. /_v2/_all_dbs pattern might work,
> See separate discussion here https://lists.apache.org/thread.html/rcc742c0fdca0363bb338b54526045720868597ea35ee6842aef174e0%40%3Cdev.couchdb.apache.org%3E
>
> Best regards,
> iilyak
>
> On 2020/04/27 15:54:11, Nick Vatamaniuc <va...@gmail.com> wrote:
> > It's good to see more activity in the thread. I thought everyone had
> > lost interest :-) Nice work, Ilya, on the prototype. I think you
> > picked what I had initially called option D and E. With the exception
> > that we don't force clients to specify a limit when a max limit is
> > configured in the settings.
> >
> > API versioning idea in principle sounds good, but can't think of a
> > clean way to do it. /_v2/_all_dbs pattern might work, but what would
> > it look like for views, perhaps /{db}/_design/{ddoc}/_v2/_view/{view}
> > is not that bad? The idea, I guess, would be to allow users the option
> > to use the old APIs to migrate their application to 4.x without having
> > to worry about rewriting everything in addition to having to configure
> > and maintain new FDB backend.
> >
> > Also if we go the API versioning route, I like the uniformity of using
> > {"items":[...]} in the response instead of {"rows": []}, or
> > {"docs":[]}. Also +1 on adding _changes in there as well.
> >
> > On the other hand, if we wanted to keep things as compatible as
> > possible, we could have a new parameter like `&use_cursor=true` for
> > view-like endpoints (excluding _all_dbs, etc), and, if that is
> > specified the user should expect the response to not return all the
> > rows sometimes and then look for a "cursor"/"bookmark" field in the
> > response. We could have an option to enforce `&use_cursor=true` usage
> > on some API points, kind of in the same vein as using a max limit
> > (option E). The point would be that user would explicitly acknowledge
> > that they are expecting this behavior. Otherwise I don't think it is a
> > good idea to keep the shape of the response the same, and only emit an
> > extra bookmark/cursor and then skip some of the rows from the results.
> >
> > Regarding not using delayed response and having configurable limits. I
> > think we'd have to see how those interact with each other. Can users
> > still set set _all_docs limit = infinity and get the current behavior?
> > They can currently (even on the prototype/fdb-layer branch). Sometimes
> > it is useful to stream all the data at once, say when doing a backup,
> > or other maintenance. Using a proper client with a cursor helper,
> > which hides the iteration loop, or a custom shell script would work,
> > but it might be nice to retain the option to be able to do it straight
> > from the API.
> >
> > There was some discussion whether we should include more or less
> > fields in the bookmark, timestamp might have issues with time skew,
> > but we could include the instance id (uuid) of the database. We do
> > that with the shard uuids currently in the update sequence. We don't
> > have to use the whole uuid but maybe just the 5-6 bytes to ensure that
> > if database is recreated and users are "streaming" data from it they
> > don't all of the sudden end up from a completely different db
> > instance. Including the update_seq is also interesting, but I think
> > the discussion so points to not do it in the first version, though it
> > would allow a user to know if, say, they backed up consistent point in
> > time snapshot of the database using a sequence of cursor requests.
> >
> > I am also +0 on returning "completed: true", I kind of like the idea.
> > Maybe we could just return a null cursor then without using yet
> > another field. "If you have a cursor you can iterate until the end, if
> > you don't you're done" kind of idea?
> >
> > Cheers,
> > -Nick
> >
> >
> > On Wed, Apr 22, 2020 at 4:19 PM Ilya Khlopotov <ii...@apache.org> wrote:
> > >
> > > Hello everyone,
> > >
> > > Based on the discussions on the thread I would like to propose a number of first steps:
> > > 1) introduce new endpoints
> > > - {db}/_all_docs/page
> > > - {db}/_all_docs/queries/page
> > > - _all_dbs/page
> > > - _dbs_info/page
> > > - {db}/_design/{ddoc}/_view/{view}/page
> > > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > - {db}/_find/page
> > >
> > > These new endpoints would act as follows:
> > > - don't use delayed responses
> > > - return object with following structure
> > > ```
> > > {
> > > "total": Total,
> > > "bookmark": base64 encoded opaque value,
> > > "completed": true | false,
> > > "update_seq": when available,
> > > "page": current page number,
> > > "items": [
> > > ]
> > > }
> > > ```
> > > - the bookmark would include following data (base64 or protobuff???):
> > > - direction
> > > - page
> > > - descending
> > > - endkey
> > > - endkey_docid
> > > - inclusive_end
> > > - startkey
> > > - startkey_docid
> > > - last_key
> > > - update_seq
> > > - timestamp
> > > ```
> > >
> > > 2) Implement per-endpoint configurable max limits
> > > ```
> > > _all_docs = 5000
> > > _all_docs/queries = 5000
> > > _all_dbs = 5000
> > > _dbs_info = 5000
> > > _view = 2500
> > > _view/queries = 2500
> > > _find = 2500
> > > ```
> > >
> > > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> > >
> > > Best regards,
> > > iilyak
> > >
> > > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > > Hello everyone,
> > > >
> > > > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> > > >
> > > > By "streaming APIs" I mean APIs which stream data in row as it gets
> > > > read from the database. These are the endpoints I was thinking of:
> > > >
> > > > _all_docs, _all_dbs, _dbs_info and query results
> > > >
> > > > I want to focus on what happens when FoundationDB transactions
> > > > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > > feeds, will crash or freeze. The reason is because the
> > > > transaction_too_old error at the end of 5 seconds is retry-able by
> > > > default, so the request handlers run again and end up shoving the
> > > > whole request down the socket again, headers and all, which is
> > > > obviously broken and not what we want.
> > > >
> > > > There are few alternatives discussed in couchdb-dev channel. I'll
> > > > present some behaviors but feel free to add more. Some ideas might
> > > > have been discounted on the IRC discussion already but I'll present
> > > > them anyway in case is sparks further conversation:
> > > >
> > > > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > > streaming the data from the next key after last emitted in the
> > > > previous transaction. Document the API behavior change that it may
> > > > present a view of the data is never a point-in-time[4] snapshot of the
> > > > DB.
> > > >
> > > > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > > don't have to change to continue using these CouchDB 4.0 endpoints
> > > > - This is the easiest to implement since it would re-use the
> > > > implementation for _changes feed (an extra option passed to the fold
> > > > function).
> > > > - Breaks API behavior if users relied on having a point-in-time[4]
> > > > snapshot view of the data.
> > > >
> > > > B) Simply end the stream. Let the users pass a `?transaction=true`
> > > > param which indicates they are aware the stream may end early and so
> > > > would have to paginate from the last emitted key with a skip=1. This
> > > > will keep the request bodies the same as current CouchDB. However, if
> > > > the users got all the data one request, they will end up wasting
> > > > another request to see if there is more data available. If they didn't
> > > > get any data they might have a too large of a skip value (see [2]) so
> > > > would have to guess different values for start/end keys. Or impose max
> > > > limit for the `skip` parameter.
> > > >
> > > > C) End the stream and add a final metadata row like a "transaction":
> > > > "timeout" at the end. That will let the user know to keep paginating
> > > > from the last key onward. This won't work for `_all_dbs` and
> > > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > > feeds and only use this for views and and _all_docs? If we like this
> > > > choice, let's think what happens for those as I couldn't come up with
> > > > anything decent there.
> > > >
> > > > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > > "key" of where the iteration stopped and the current "skip" and
> > > > "limit" params, which would keep decreasing. Then user would pass
> > > > those in "start_key=..." in the next request along with the limit and
> > > > skip params. So something like "continuation":{"skip":599, "limit":5,
> > > > "key":"..."}. This has the same issue with array results for
> > > > `_all_dbs` and `_dbs_info`[3].
> > > >
> > > > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > > there such that response time is likely to fit in one transaction.
> > > > This could be tricky as different runtime environments will have
> > > > different characteristics. Also, if the timeout happens there isn't a
> > > > a nice way to send an HTTP error since we already sent the 200
> > > > response. The downside is that this might break how some users use the
> > > > API, if say the are using large skips and limits already. Perhaps here
> > > > we do both B and D, such that if users want transactional behavior,
> > > > they specify that `transaction=true` param and only then we enforce
> > > > low limit and skip maximums.
> > > >
> > > > F) At least for `_all_docs` it seems providing a point-in-time
> > > > snapshot view doesn't necessarily need to be tied to transaction
> > > > boundaries. We could check the update sequence of the database at the
> > > > start of the next transaction and if it hasn't changed we can continue
> > > > emitting a consistent view. This can apply to C and D and would just
> > > > determine when the stream ends. If there are no writes happening to
> > > > the db, this could potential streams all the data just like option A
> > > > would do. Not entirely sure if this would work for views.
> > > >
> > > > So what do we think? I can see different combinations of options here,
> > > > maybe even different for each API point. For example `_all_dbs`,
> > > > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > > have parameters to do F, etc.
> > > >
> > > > Cheers,
> > > > -Nick
> > > >
> > > > Some footnotes:
> > > >
> > > > [1] _changes feeds is the only one that works currently. It behaves as
> > > > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > > > That is, we continue streaming the data by resetting the transaction
> > > > object and restarting from the last emitted key (db sequence in this
> > > > case). However, because the transaction restarts if a document is
> > > > updated while the streaming take place, it may appear in the _changes
> > > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > > have to document it, since previously we presented this point-in-time
> > > > snapshot of the database from when we started streaming.
> > > >
> > > > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > > currently support efficient offsets for key selectors
> > > > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > > > we implemented skip by iterating over the data. This means that a skip
> > > > of say 100000 could keep timing out the transaction without yielding
> > > > any data.
> > > >
> > > > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > > obvious place to insert a last metadata row.
> > > >
> > > > [4] For example they have a constraint that documents "a" and "z"
> > > > cannot both be in the database at the same time. But when iterating
> > > > it's possible that "a" was there at the start. Then by the end, "a"
> > > > was removed and "z" added, so both "a" and "z" would appear in the
> > > > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > > "relaxed" constrains:
> > > > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > > >
> >
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
Hi Nick,
Thank you for extensive answer.
> API versioning idea in principle sounds good, but can't think of a
> clean way to do it. /_v2/_all_dbs pattern might work,
See separate discussion here https://lists.apache.org/thread.html/rcc742c0fdca0363bb338b54526045720868597ea35ee6842aef174e0%40%3Cdev.couchdb.apache.org%3E
Best regards,
iilyak
On 2020/04/27 15:54:11, Nick Vatamaniuc <va...@gmail.com> wrote:
> It's good to see more activity in the thread. I thought everyone had
> lost interest :-) Nice work, Ilya, on the prototype. I think you
> picked what I had initially called option D and E. With the exception
> that we don't force clients to specify a limit when a max limit is
> configured in the settings.
>
> API versioning idea in principle sounds good, but can't think of a
> clean way to do it. /_v2/_all_dbs pattern might work, but what would
> it look like for views, perhaps /{db}/_design/{ddoc}/_v2/_view/{view}
> is not that bad? The idea, I guess, would be to allow users the option
> to use the old APIs to migrate their application to 4.x without having
> to worry about rewriting everything in addition to having to configure
> and maintain new FDB backend.
>
> Also if we go the API versioning route, I like the uniformity of using
> {"items":[...]} in the response instead of {"rows": []}, or
> {"docs":[]}. Also +1 on adding _changes in there as well.
>
> On the other hand, if we wanted to keep things as compatible as
> possible, we could have a new parameter like `&use_cursor=true` for
> view-like endpoints (excluding _all_dbs, etc), and, if that is
> specified the user should expect the response to not return all the
> rows sometimes and then look for a "cursor"/"bookmark" field in the
> response. We could have an option to enforce `&use_cursor=true` usage
> on some API points, kind of in the same vein as using a max limit
> (option E). The point would be that user would explicitly acknowledge
> that they are expecting this behavior. Otherwise I don't think it is a
> good idea to keep the shape of the response the same, and only emit an
> extra bookmark/cursor and then skip some of the rows from the results.
>
> Regarding not using delayed response and having configurable limits. I
> think we'd have to see how those interact with each other. Can users
> still set set _all_docs limit = infinity and get the current behavior?
> They can currently (even on the prototype/fdb-layer branch). Sometimes
> it is useful to stream all the data at once, say when doing a backup,
> or other maintenance. Using a proper client with a cursor helper,
> which hides the iteration loop, or a custom shell script would work,
> but it might be nice to retain the option to be able to do it straight
> from the API.
>
> There was some discussion whether we should include more or less
> fields in the bookmark, timestamp might have issues with time skew,
> but we could include the instance id (uuid) of the database. We do
> that with the shard uuids currently in the update sequence. We don't
> have to use the whole uuid but maybe just the 5-6 bytes to ensure that
> if database is recreated and users are "streaming" data from it they
> don't all of the sudden end up from a completely different db
> instance. Including the update_seq is also interesting, but I think
> the discussion so points to not do it in the first version, though it
> would allow a user to know if, say, they backed up consistent point in
> time snapshot of the database using a sequence of cursor requests.
>
> I am also +0 on returning "completed: true", I kind of like the idea.
> Maybe we could just return a null cursor then without using yet
> another field. "If you have a cursor you can iterate until the end, if
> you don't you're done" kind of idea?
>
> Cheers,
> -Nick
>
>
> On Wed, Apr 22, 2020 at 4:19 PM Ilya Khlopotov <ii...@apache.org> wrote:
> >
> > Hello everyone,
> >
> > Based on the discussions on the thread I would like to propose a number of first steps:
> > 1) introduce new endpoints
> > - {db}/_all_docs/page
> > - {db}/_all_docs/queries/page
> > - _all_dbs/page
> > - _dbs_info/page
> > - {db}/_design/{ddoc}/_view/{view}/page
> > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > - {db}/_find/page
> >
> > These new endpoints would act as follows:
> > - don't use delayed responses
> > - return object with following structure
> > ```
> > {
> > "total": Total,
> > "bookmark": base64 encoded opaque value,
> > "completed": true | false,
> > "update_seq": when available,
> > "page": current page number,
> > "items": [
> > ]
> > }
> > ```
> > - the bookmark would include following data (base64 or protobuff???):
> > - direction
> > - page
> > - descending
> > - endkey
> > - endkey_docid
> > - inclusive_end
> > - startkey
> > - startkey_docid
> > - last_key
> > - update_seq
> > - timestamp
> > ```
> >
> > 2) Implement per-endpoint configurable max limits
> > ```
> > _all_docs = 5000
> > _all_docs/queries = 5000
> > _all_dbs = 5000
> > _dbs_info = 5000
> > _view = 2500
> > _view/queries = 2500
> > _find = 2500
> > ```
> >
> > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> >
> > Best regards,
> > iilyak
> >
> > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > Hello everyone,
> > >
> > > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> > >
> > > By "streaming APIs" I mean APIs which stream data in row as it gets
> > > read from the database. These are the endpoints I was thinking of:
> > >
> > > _all_docs, _all_dbs, _dbs_info and query results
> > >
> > > I want to focus on what happens when FoundationDB transactions
> > > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > feeds, will crash or freeze. The reason is because the
> > > transaction_too_old error at the end of 5 seconds is retry-able by
> > > default, so the request handlers run again and end up shoving the
> > > whole request down the socket again, headers and all, which is
> > > obviously broken and not what we want.
> > >
> > > There are few alternatives discussed in couchdb-dev channel. I'll
> > > present some behaviors but feel free to add more. Some ideas might
> > > have been discounted on the IRC discussion already but I'll present
> > > them anyway in case is sparks further conversation:
> > >
> > > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > streaming the data from the next key after last emitted in the
> > > previous transaction. Document the API behavior change that it may
> > > present a view of the data is never a point-in-time[4] snapshot of the
> > > DB.
> > >
> > > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > don't have to change to continue using these CouchDB 4.0 endpoints
> > > - This is the easiest to implement since it would re-use the
> > > implementation for _changes feed (an extra option passed to the fold
> > > function).
> > > - Breaks API behavior if users relied on having a point-in-time[4]
> > > snapshot view of the data.
> > >
> > > B) Simply end the stream. Let the users pass a `?transaction=true`
> > > param which indicates they are aware the stream may end early and so
> > > would have to paginate from the last emitted key with a skip=1. This
> > > will keep the request bodies the same as current CouchDB. However, if
> > > the users got all the data one request, they will end up wasting
> > > another request to see if there is more data available. If they didn't
> > > get any data they might have a too large of a skip value (see [2]) so
> > > would have to guess different values for start/end keys. Or impose max
> > > limit for the `skip` parameter.
> > >
> > > C) End the stream and add a final metadata row like a "transaction":
> > > "timeout" at the end. That will let the user know to keep paginating
> > > from the last key onward. This won't work for `_all_dbs` and
> > > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > feeds and only use this for views and and _all_docs? If we like this
> > > choice, let's think what happens for those as I couldn't come up with
> > > anything decent there.
> > >
> > > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > "key" of where the iteration stopped and the current "skip" and
> > > "limit" params, which would keep decreasing. Then user would pass
> > > those in "start_key=..." in the next request along with the limit and
> > > skip params. So something like "continuation":{"skip":599, "limit":5,
> > > "key":"..."}. This has the same issue with array results for
> > > `_all_dbs` and `_dbs_info`[3].
> > >
> > > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > there such that response time is likely to fit in one transaction.
> > > This could be tricky as different runtime environments will have
> > > different characteristics. Also, if the timeout happens there isn't a
> > > a nice way to send an HTTP error since we already sent the 200
> > > response. The downside is that this might break how some users use the
> > > API, if say the are using large skips and limits already. Perhaps here
> > > we do both B and D, such that if users want transactional behavior,
> > > they specify that `transaction=true` param and only then we enforce
> > > low limit and skip maximums.
> > >
> > > F) At least for `_all_docs` it seems providing a point-in-time
> > > snapshot view doesn't necessarily need to be tied to transaction
> > > boundaries. We could check the update sequence of the database at the
> > > start of the next transaction and if it hasn't changed we can continue
> > > emitting a consistent view. This can apply to C and D and would just
> > > determine when the stream ends. If there are no writes happening to
> > > the db, this could potential streams all the data just like option A
> > > would do. Not entirely sure if this would work for views.
> > >
> > > So what do we think? I can see different combinations of options here,
> > > maybe even different for each API point. For example `_all_dbs`,
> > > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > have parameters to do F, etc.
> > >
> > > Cheers,
> > > -Nick
> > >
> > > Some footnotes:
> > >
> > > [1] _changes feeds is the only one that works currently. It behaves as
> > > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > > That is, we continue streaming the data by resetting the transaction
> > > object and restarting from the last emitted key (db sequence in this
> > > case). However, because the transaction restarts if a document is
> > > updated while the streaming take place, it may appear in the _changes
> > > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > have to document it, since previously we presented this point-in-time
> > > snapshot of the database from when we started streaming.
> > >
> > > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > currently support efficient offsets for key selectors
> > > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > > we implemented skip by iterating over the data. This means that a skip
> > > of say 100000 could keep timing out the transaction without yielding
> > > any data.
> > >
> > > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > obvious place to insert a last metadata row.
> > >
> > > [4] For example they have a constraint that documents "a" and "z"
> > > cannot both be in the database at the same time. But when iterating
> > > it's possible that "a" was there at the start. Then by the end, "a"
> > > was removed and "z" added, so both "a" and "z" would appear in the
> > > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > "relaxed" constrains:
> > > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Nick Vatamaniuc <va...@gmail.com>.
It's good to see more activity in the thread. I thought everyone had
lost interest :-) Nice work, Ilya, on the prototype. I think you
picked what I had initially called option D and E. With the exception
that we don't force clients to specify a limit when a max limit is
configured in the settings.
API versioning idea in principle sounds good, but can't think of a
clean way to do it. /_v2/_all_dbs pattern might work, but what would
it look like for views, perhaps /{db}/_design/{ddoc}/_v2/_view/{view}
is not that bad? The idea, I guess, would be to allow users the option
to use the old APIs to migrate their application to 4.x without having
to worry about rewriting everything in addition to having to configure
and maintain new FDB backend.
Also if we go the API versioning route, I like the uniformity of using
{"items":[...]} in the response instead of {"rows": []}, or
{"docs":[]}. Also +1 on adding _changes in there as well.
On the other hand, if we wanted to keep things as compatible as
possible, we could have a new parameter like `&use_cursor=true` for
view-like endpoints (excluding _all_dbs, etc), and, if that is
specified the user should expect the response to not return all the
rows sometimes and then look for a "cursor"/"bookmark" field in the
response. We could have an option to enforce `&use_cursor=true` usage
on some API points, kind of in the same vein as using a max limit
(option E). The point would be that user would explicitly acknowledge
that they are expecting this behavior. Otherwise I don't think it is a
good idea to keep the shape of the response the same, and only emit an
extra bookmark/cursor and then skip some of the rows from the results.
Regarding not using delayed response and having configurable limits. I
think we'd have to see how those interact with each other. Can users
still set set _all_docs limit = infinity and get the current behavior?
They can currently (even on the prototype/fdb-layer branch). Sometimes
it is useful to stream all the data at once, say when doing a backup,
or other maintenance. Using a proper client with a cursor helper,
which hides the iteration loop, or a custom shell script would work,
but it might be nice to retain the option to be able to do it straight
from the API.
There was some discussion whether we should include more or less
fields in the bookmark, timestamp might have issues with time skew,
but we could include the instance id (uuid) of the database. We do
that with the shard uuids currently in the update sequence. We don't
have to use the whole uuid but maybe just the 5-6 bytes to ensure that
if database is recreated and users are "streaming" data from it they
don't all of the sudden end up from a completely different db
instance. Including the update_seq is also interesting, but I think
the discussion so points to not do it in the first version, though it
would allow a user to know if, say, they backed up consistent point in
time snapshot of the database using a sequence of cursor requests.
I am also +0 on returning "completed: true", I kind of like the idea.
Maybe we could just return a null cursor then without using yet
another field. "If you have a cursor you can iterate until the end, if
you don't you're done" kind of idea?
Cheers,
-Nick
On Wed, Apr 22, 2020 at 4:19 PM Ilya Khlopotov <ii...@apache.org> wrote:
>
> Hello everyone,
>
> Based on the discussions on the thread I would like to propose a number of first steps:
> 1) introduce new endpoints
> - {db}/_all_docs/page
> - {db}/_all_docs/queries/page
> - _all_dbs/page
> - _dbs_info/page
> - {db}/_design/{ddoc}/_view/{view}/page
> - {db}/_design/{ddoc}/_view/{view}/queries/page
> - {db}/_find/page
>
> These new endpoints would act as follows:
> - don't use delayed responses
> - return object with following structure
> ```
> {
> "total": Total,
> "bookmark": base64 encoded opaque value,
> "completed": true | false,
> "update_seq": when available,
> "page": current page number,
> "items": [
> ]
> }
> ```
> - the bookmark would include following data (base64 or protobuff???):
> - direction
> - page
> - descending
> - endkey
> - endkey_docid
> - inclusive_end
> - startkey
> - startkey_docid
> - last_key
> - update_seq
> - timestamp
> ```
>
> 2) Implement per-endpoint configurable max limits
> ```
> _all_docs = 5000
> _all_docs/queries = 5000
> _all_dbs = 5000
> _dbs_info = 5000
> _view = 2500
> _view/queries = 2500
> _find = 2500
> ```
>
> Latter (after few years) CouchDB would deprecate and remove old endpoints.
>
> Best regards,
> iilyak
>
> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > Hello everyone,
> >
> > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> >
> > By "streaming APIs" I mean APIs which stream data in row as it gets
> > read from the database. These are the endpoints I was thinking of:
> >
> > _all_docs, _all_dbs, _dbs_info and query results
> >
> > I want to focus on what happens when FoundationDB transactions
> > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > feeds, will crash or freeze. The reason is because the
> > transaction_too_old error at the end of 5 seconds is retry-able by
> > default, so the request handlers run again and end up shoving the
> > whole request down the socket again, headers and all, which is
> > obviously broken and not what we want.
> >
> > There are few alternatives discussed in couchdb-dev channel. I'll
> > present some behaviors but feel free to add more. Some ideas might
> > have been discounted on the IRC discussion already but I'll present
> > them anyway in case is sparks further conversation:
> >
> > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > streaming the data from the next key after last emitted in the
> > previous transaction. Document the API behavior change that it may
> > present a view of the data is never a point-in-time[4] snapshot of the
> > DB.
> >
> > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > don't have to change to continue using these CouchDB 4.0 endpoints
> > - This is the easiest to implement since it would re-use the
> > implementation for _changes feed (an extra option passed to the fold
> > function).
> > - Breaks API behavior if users relied on having a point-in-time[4]
> > snapshot view of the data.
> >
> > B) Simply end the stream. Let the users pass a `?transaction=true`
> > param which indicates they are aware the stream may end early and so
> > would have to paginate from the last emitted key with a skip=1. This
> > will keep the request bodies the same as current CouchDB. However, if
> > the users got all the data one request, they will end up wasting
> > another request to see if there is more data available. If they didn't
> > get any data they might have a too large of a skip value (see [2]) so
> > would have to guess different values for start/end keys. Or impose max
> > limit for the `skip` parameter.
> >
> > C) End the stream and add a final metadata row like a "transaction":
> > "timeout" at the end. That will let the user know to keep paginating
> > from the last key onward. This won't work for `_all_dbs` and
> > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > feeds and only use this for views and and _all_docs? If we like this
> > choice, let's think what happens for those as I couldn't come up with
> > anything decent there.
> >
> > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > "key" of where the iteration stopped and the current "skip" and
> > "limit" params, which would keep decreasing. Then user would pass
> > those in "start_key=..." in the next request along with the limit and
> > skip params. So something like "continuation":{"skip":599, "limit":5,
> > "key":"..."}. This has the same issue with array results for
> > `_all_dbs` and `_dbs_info`[3].
> >
> > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > there such that response time is likely to fit in one transaction.
> > This could be tricky as different runtime environments will have
> > different characteristics. Also, if the timeout happens there isn't a
> > a nice way to send an HTTP error since we already sent the 200
> > response. The downside is that this might break how some users use the
> > API, if say the are using large skips and limits already. Perhaps here
> > we do both B and D, such that if users want transactional behavior,
> > they specify that `transaction=true` param and only then we enforce
> > low limit and skip maximums.
> >
> > F) At least for `_all_docs` it seems providing a point-in-time
> > snapshot view doesn't necessarily need to be tied to transaction
> > boundaries. We could check the update sequence of the database at the
> > start of the next transaction and if it hasn't changed we can continue
> > emitting a consistent view. This can apply to C and D and would just
> > determine when the stream ends. If there are no writes happening to
> > the db, this could potential streams all the data just like option A
> > would do. Not entirely sure if this would work for views.
> >
> > So what do we think? I can see different combinations of options here,
> > maybe even different for each API point. For example `_all_dbs`,
> > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > have parameters to do F, etc.
> >
> > Cheers,
> > -Nick
> >
> > Some footnotes:
> >
> > [1] _changes feeds is the only one that works currently. It behaves as
> > per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > That is, we continue streaming the data by resetting the transaction
> > object and restarting from the last emitted key (db sequence in this
> > case). However, because the transaction restarts if a document is
> > updated while the streaming take place, it may appear in the _changes
> > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > have to document it, since previously we presented this point-in-time
> > snapshot of the database from when we started streaming.
> >
> > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > currently support efficient offsets for key selectors
> > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > we implemented skip by iterating over the data. This means that a skip
> > of say 100000 could keep timing out the transaction without yielding
> > any data.
> >
> > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > obvious place to insert a last metadata row.
> >
> > [4] For example they have a constraint that documents "a" and "z"
> > cannot both be in the database at the same time. But when iterating
> > it's possible that "a" was there at the start. Then by the end, "a"
> > was removed and "z" added, so both "a" and "z" would appear in the
> > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > "relaxed" constrains:
> > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
> All the fields in the bookmark make sense except timestamp. Why would it
> matter if the timestamp is old? What happens if a node's time is an hour
> behind another node?
Bookmarks are not permalinks. The exact behavior would depend on the type of pagination.
# offset based pagination
There will be page drifts when data inserted or removed while we traversing. The probability of this anomaly would increase if we use older bookmarks.
# last key based pagination
Uses the last key from the previous page. This type of pagination doesn't work if the key we hold on to was removed between the pages. Luckily we can avoid this problem by using:
- FDB_KEYSEL_FIRST_GREATER_OR_EQUAL
- FDB_KEYSEL_LAST_LESS_OR_EQUAL
I don't know whether this ability is exposed in erlfdb or not.
On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
> I agree with Bob that page doesn't make sense as an endpoint. I'm also
> rubbish with naming. The best I could come up with is replacing page with
> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> All the fields in the bookmark make sense except timestamp. Why would it
> matter if the timestamp is old? What happens if a node's time is an hour
> behind another node?
>
>
> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
>
> > - page is to provide some notion of progress for user
> > - timestamp - I was thinking that we should drop requests if user would
> > try to pass bookmark created an hour ago.
> >
> > On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
> > > "page" and "page number" are odd to me as these don't exist as concepts,
> > I'd rather not invent them. I note there's no mention of page size, which
> > makes "page number" very vague.
> > >
> > > What is "timestamp" in the bookmark and what effect does it have when
> > the bookmark is passed back in?
> > >
> > > I guess, why does the bookmark include so much extraneous data? Items
> > that are not needed to find the fdb key to begin the next response from.
> > >
> > >
> > > > On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > Based on the discussions on the thread I would like to propose a
> > number of first steps:
> > > > 1) introduce new endpoints
> > > > - {db}/_all_docs/page
> > > > - {db}/_all_docs/queries/page
> > > > - _all_dbs/page
> > > > - _dbs_info/page
> > > > - {db}/_design/{ddoc}/_view/{view}/page
> > > > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > > - {db}/_find/page
> > > >
> > > > These new endpoints would act as follows:
> > > > - don't use delayed responses
> > > > - return object with following structure
> > > > ```
> > > > {
> > > > "total": Total,
> > > > "bookmark": base64 encoded opaque value,
> > > > "completed": true | false,
> > > > "update_seq": when available,
> > > > "page": current page number,
> > > > "items": [
> > > > ]
> > > > }
> > > > ```
> > > > - the bookmark would include following data (base64 or protobuff???):
> > > > - direction
> > > > - page
> > > > - descending
> > > > - endkey
> > > > - endkey_docid
> > > > - inclusive_end
> > > > - startkey
> > > > - startkey_docid
> > > > - last_key
> > > > - update_seq
> > > > - timestamp
> > > > ```
> > > >
> > > > 2) Implement per-endpoint configurable max limits
> > > > ```
> > > > _all_docs = 5000
> > > > _all_docs/queries = 5000
> > > > _all_dbs = 5000
> > > > _dbs_info = 5000
> > > > _view = 2500
> > > > _view/queries = 2500
> > > > _find = 2500
> > > > ```
> > > >
> > > > Latter (after few years) CouchDB would deprecate and remove old
> > endpoints.
> > > >
> > > > Best regards,
> > > > iilyak
> > > >
> > > > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > >> Hello everyone,
> > > >>
> > > >> I'd like to discuss the shape and behavior of streaming APIs for
> > CouchDB 4.x
> > > >>
> > > >> By "streaming APIs" I mean APIs which stream data in row as it gets
> > > >> read from the database. These are the endpoints I was thinking of:
> > > >>
> > > >> _all_docs, _all_dbs, _dbs_info and query results
> > > >>
> > > >> I want to focus on what happens when FoundationDB transactions
> > > >> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > >> feeds, will crash or freeze. The reason is because the
> > > >> transaction_too_old error at the end of 5 seconds is retry-able by
> > > >> default, so the request handlers run again and end up shoving the
> > > >> whole request down the socket again, headers and all, which is
> > > >> obviously broken and not what we want.
> > > >>
> > > >> There are few alternatives discussed in couchdb-dev channel. I'll
> > > >> present some behaviors but feel free to add more. Some ideas might
> > > >> have been discounted on the IRC discussion already but I'll present
> > > >> them anyway in case is sparks further conversation:
> > > >>
> > > >> A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > >> streaming the data from the next key after last emitted in the
> > > >> previous transaction. Document the API behavior change that it may
> > > >> present a view of the data is never a point-in-time[4] snapshot of the
> > > >> DB.
> > > >>
> > > >> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > >> don't have to change to continue using these CouchDB 4.0 endpoints
> > > >> - This is the easiest to implement since it would re-use the
> > > >> implementation for _changes feed (an extra option passed to the fold
> > > >> function).
> > > >> - Breaks API behavior if users relied on having a point-in-time[4]
> > > >> snapshot view of the data.
> > > >>
> > > >> B) Simply end the stream. Let the users pass a `?transaction=true`
> > > >> param which indicates they are aware the stream may end early and so
> > > >> would have to paginate from the last emitted key with a skip=1. This
> > > >> will keep the request bodies the same as current CouchDB. However, if
> > > >> the users got all the data one request, they will end up wasting
> > > >> another request to see if there is more data available. If they didn't
> > > >> get any data they might have a too large of a skip value (see [2]) so
> > > >> would have to guess different values for start/end keys. Or impose max
> > > >> limit for the `skip` parameter.
> > > >>
> > > >> C) End the stream and add a final metadata row like a "transaction":
> > > >> "timeout" at the end. That will let the user know to keep paginating
> > > >> from the last key onward. This won't work for `_all_dbs` and
> > > >> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > >> feeds and only use this for views and and _all_docs? If we like this
> > > >> choice, let's think what happens for those as I couldn't come up with
> > > >> anything decent there.
> > > >>
> > > >> D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > >> "key" of where the iteration stopped and the current "skip" and
> > > >> "limit" params, which would keep decreasing. Then user would pass
> > > >> those in "start_key=..." in the next request along with the limit and
> > > >> skip params. So something like "continuation":{"skip":599, "limit":5,
> > > >> "key":"..."}. This has the same issue with array results for
> > > >> `_all_dbs` and `_dbs_info`[3].
> > > >>
> > > >> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > >> there such that response time is likely to fit in one transaction.
> > > >> This could be tricky as different runtime environments will have
> > > >> different characteristics. Also, if the timeout happens there isn't a
> > > >> a nice way to send an HTTP error since we already sent the 200
> > > >> response. The downside is that this might break how some users use the
> > > >> API, if say the are using large skips and limits already. Perhaps here
> > > >> we do both B and D, such that if users want transactional behavior,
> > > >> they specify that `transaction=true` param and only then we enforce
> > > >> low limit and skip maximums.
> > > >>
> > > >> F) At least for `_all_docs` it seems providing a point-in-time
> > > >> snapshot view doesn't necessarily need to be tied to transaction
> > > >> boundaries. We could check the update sequence of the database at the
> > > >> start of the next transaction and if it hasn't changed we can continue
> > > >> emitting a consistent view. This can apply to C and D and would just
> > > >> determine when the stream ends. If there are no writes happening to
> > > >> the db, this could potential streams all the data just like option A
> > > >> would do. Not entirely sure if this would work for views.
> > > >>
> > > >> So what do we think? I can see different combinations of options here,
> > > >> maybe even different for each API point. For example `_all_dbs`,
> > > >> `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > >> have parameters to do F, etc.
> > > >>
> > > >> Cheers,
> > > >> -Nick
> > > >>
> > > >> Some footnotes:
> > > >>
> > > >> [1] _changes feeds is the only one that works currently. It behaves as
> > > >> per RFC
> > https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> > .
> > > >> That is, we continue streaming the data by resetting the transaction
> > > >> object and restarting from the last emitted key (db sequence in this
> > > >> case). However, because the transaction restarts if a document is
> > > >> updated while the streaming take place, it may appear in the _changes
> > > >> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > >> have to document it, since previously we presented this point-in-time
> > > >> snapshot of the database from when we started streaming.
> > > >>
> > > >> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > >> currently support efficient offsets for key selectors
> > > >> (
> > https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> > )
> > > >> we implemented skip by iterating over the data. This means that a skip
> > > >> of say 100000 could keep timing out the transaction without yielding
> > > >> any data.
> > > >>
> > > >> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > >> obvious place to insert a last metadata row.
> > > >>
> > > >> [4] For example they have a constraint that documents "a" and "z"
> > > >> cannot both be in the database at the same time. But when iterating
> > > >> it's possible that "a" was there at the start. Then by the end, "a"
> > > >> was removed and "z" added, so both "a" and "z" would appear in the
> > > >> emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > >> "relaxed" constrains:
> > > >>
> > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
> https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
Very good article. My PoC experiment is in fact implementation of a cursor based pagination.
Event though the bookmark encodes all non default values of mrargs the algorithm only uses:
- limit - doesn't change
- start_key - updated for every bookmark as we iterate
- end_key - doesn't change
- direction - doesn't change
Best regards,
iilyak
On 2020/04/23 18:02:34, Joan Touzet <wo...@apache.org> wrote:
> I realise this is bikeshedding, but I guess that's kind of the point...
> Everything below is my opinion, not "fact."
>
> It's unfortunate we need a new endpoint for all of this. In a vacuum I
> might have just suggested we use the semantics we already have, perhaps
> with ?from= instead of ?since= .
>
> "page" only works if the size of a page is well known, either by server
> preference or directly in the URL. If I ask for:
>
> GET /{db}/_all_docs?limit=20&page=3
>
> I know that I'm always going to get document 41 through 60 in the
> default collation order.
>
> There's a *fantastic* summary of examples from popular REST APIs here:
>
>
> https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
>
> We are *pretty close* to what a cursor means in those other examples,
> except for the fact that our cursor can go stale/invalid after a short time.
>
> Bob, could you be a bit more detailed in your explanation how our
> definition isn't close to these? Or did you mean SQL CURSOR (which is
> something entirely different?) If so, I'm fine with this being a REST
> API cursor - something clearly distinct.
>
> I come back to wanting to preserve the existing endpoint syntax and
> naming, without new endpoints, but specifying this new FDB token via
> ?cursor= and this being the trigger for the new behaviour. At some
> point, we simply stop accepting ?since= tokens. This seems inline with
> other popular REST APIs.
>
> -Joan "still sick and not sleeping right" Touzet
>
>
> On 2020-04-23 12:30, Robert Newson wrote:
> > cursor has established meaning in other databases and ours would not be very close to them. I don’t think it’s a good idea.
> >
> > B.
> >
> >> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
> >>
> >>
> >>>
> >>> The best I could come up with is replacing page with
> >>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> >> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
> >>
> >>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
> >>> I agree with Bob that page doesn't make sense as an endpoint. I'm also
> >>> rubbish with naming. The best I could come up with is replacing page with
> >>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> >>> All the fields in the bookmark make sense except timestamp. Why would it
> >>> matter if the timestamp is old? What happens if a node's time is an hour
> >>> behind another node?
> >>>
> >>>
> >>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
> >>>>
> >>>> - page is to provide some notion of progress for user
> >>>> - timestamp - I was thinking that we should drop requests if user would
> >>>> try to pass bookmark created an hour ago.
> >>>>
> >>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
> >>>>> "page" and "page number" are odd to me as these don't exist as concepts,
> >>>> I'd rather not invent them. I note there's no mention of page size, which
> >>>> makes "page number" very vague.
> >>>>>
> >>>>> What is "timestamp" in the bookmark and what effect does it have when
> >>>> the bookmark is passed back in?
> >>>>>
> >>>>> I guess, why does the bookmark include so much extraneous data? Items
> >>>> that are not needed to find the fdb key to begin the next response from.
> >>>>>
> >>>>>
> >>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
> >>>>>>
> >>>>>> Hello everyone,
> >>>>>>
> >>>>>> Based on the discussions on the thread I would like to propose a
> >>>> number of first steps:
> >>>>>> 1) introduce new endpoints
> >>>>>> - {db}/_all_docs/page
> >>>>>> - {db}/_all_docs/queries/page
> >>>>>> - _all_dbs/page
> >>>>>> - _dbs_info/page
> >>>>>> - {db}/_design/{ddoc}/_view/{view}/page
> >>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
> >>>>>> - {db}/_find/page
> >>>>>>
> >>>>>> These new endpoints would act as follows:
> >>>>>> - don't use delayed responses
> >>>>>> - return object with following structure
> >>>>>> ```
> >>>>>> {
> >>>>>> "total": Total,
> >>>>>> "bookmark": base64 encoded opaque value,
> >>>>>> "completed": true | false,
> >>>>>> "update_seq": when available,
> >>>>>> "page": current page number,
> >>>>>> "items": [
> >>>>>> ]
> >>>>>> }
> >>>>>> ```
> >>>>>> - the bookmark would include following data (base64 or protobuff???):
> >>>>>> - direction
> >>>>>> - page
> >>>>>> - descending
> >>>>>> - endkey
> >>>>>> - endkey_docid
> >>>>>> - inclusive_end
> >>>>>> - startkey
> >>>>>> - startkey_docid
> >>>>>> - last_key
> >>>>>> - update_seq
> >>>>>> - timestamp
> >>>>>> ```
> >>>>>>
> >>>>>> 2) Implement per-endpoint configurable max limits
> >>>>>> ```
> >>>>>> _all_docs = 5000
> >>>>>> _all_docs/queries = 5000
> >>>>>> _all_dbs = 5000
> >>>>>> _dbs_info = 5000
> >>>>>> _view = 2500
> >>>>>> _view/queries = 2500
> >>>>>> _find = 2500
> >>>>>> ```
> >>>>>>
> >>>>>> Latter (after few years) CouchDB would deprecate and remove old
> >>>> endpoints.
> >>>>>>
> >>>>>> Best regards,
> >>>>>> iilyak
> >>>>>>
> >>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> >>>>>>> Hello everyone,
> >>>>>>>
> >>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
> >>>> CouchDB 4.x
> >>>>>>>
> >>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
> >>>>>>> read from the database. These are the endpoints I was thinking of:
> >>>>>>>
> >>>>>>> _all_docs, _all_dbs, _dbs_info and query results
> >>>>>>>
> >>>>>>> I want to focus on what happens when FoundationDB transactions
> >>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> >>>>>>> feeds, will crash or freeze. The reason is because the
> >>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
> >>>>>>> default, so the request handlers run again and end up shoving the
> >>>>>>> whole request down the socket again, headers and all, which is
> >>>>>>> obviously broken and not what we want.
> >>>>>>>
> >>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
> >>>>>>> present some behaviors but feel free to add more. Some ideas might
> >>>>>>> have been discounted on the IRC discussion already but I'll present
> >>>>>>> them anyway in case is sparks further conversation:
> >>>>>>>
> >>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
> >>>>>>> streaming the data from the next key after last emitted in the
> >>>>>>> previous transaction. Document the API behavior change that it may
> >>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
> >>>>>>> DB.
> >>>>>>>
> >>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> >>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
> >>>>>>> - This is the easiest to implement since it would re-use the
> >>>>>>> implementation for _changes feed (an extra option passed to the fold
> >>>>>>> function).
> >>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
> >>>>>>> snapshot view of the data.
> >>>>>>>
> >>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
> >>>>>>> param which indicates they are aware the stream may end early and so
> >>>>>>> would have to paginate from the last emitted key with a skip=1. This
> >>>>>>> will keep the request bodies the same as current CouchDB. However, if
> >>>>>>> the users got all the data one request, they will end up wasting
> >>>>>>> another request to see if there is more data available. If they didn't
> >>>>>>> get any data they might have a too large of a skip value (see [2]) so
> >>>>>>> would have to guess different values for start/end keys. Or impose max
> >>>>>>> limit for the `skip` parameter.
> >>>>>>>
> >>>>>>> C) End the stream and add a final metadata row like a "transaction":
> >>>>>>> "timeout" at the end. That will let the user know to keep paginating
> >>>>>>> from the last key onward. This won't work for `_all_dbs` and
> >>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> >>>>>>> feeds and only use this for views and and _all_docs? If we like this
> >>>>>>> choice, let's think what happens for those as I couldn't come up with
> >>>>>>> anything decent there.
> >>>>>>>
> >>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
> >>>>>>> "key" of where the iteration stopped and the current "skip" and
> >>>>>>> "limit" params, which would keep decreasing. Then user would pass
> >>>>>>> those in "start_key=..." in the next request along with the limit and
> >>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
> >>>>>>> "key":"..."}. This has the same issue with array results for
> >>>>>>> `_all_dbs` and `_dbs_info`[3].
> >>>>>>>
> >>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> >>>>>>> there such that response time is likely to fit in one transaction.
> >>>>>>> This could be tricky as different runtime environments will have
> >>>>>>> different characteristics. Also, if the timeout happens there isn't a
> >>>>>>> a nice way to send an HTTP error since we already sent the 200
> >>>>>>> response. The downside is that this might break how some users use the
> >>>>>>> API, if say the are using large skips and limits already. Perhaps here
> >>>>>>> we do both B and D, such that if users want transactional behavior,
> >>>>>>> they specify that `transaction=true` param and only then we enforce
> >>>>>>> low limit and skip maximums.
> >>>>>>>
> >>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
> >>>>>>> snapshot view doesn't necessarily need to be tied to transaction
> >>>>>>> boundaries. We could check the update sequence of the database at the
> >>>>>>> start of the next transaction and if it hasn't changed we can continue
> >>>>>>> emitting a consistent view. This can apply to C and D and would just
> >>>>>>> determine when the stream ends. If there are no writes happening to
> >>>>>>> the db, this could potential streams all the data just like option A
> >>>>>>> would do. Not entirely sure if this would work for views.
> >>>>>>>
> >>>>>>> So what do we think? I can see different combinations of options here,
> >>>>>>> maybe even different for each API point. For example `_all_dbs`,
> >>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
> >>>>>>> have parameters to do F, etc.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> -Nick
> >>>>>>>
> >>>>>>> Some footnotes:
> >>>>>>>
> >>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
> >>>>>>> per RFC
> >>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> >>>> .
> >>>>>>> That is, we continue streaming the data by resetting the transaction
> >>>>>>> object and restarting from the last emitted key (db sequence in this
> >>>>>>> case). However, because the transaction restarts if a document is
> >>>>>>> updated while the streaming take place, it may appear in the _changes
> >>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> >>>>>>> have to document it, since previously we presented this point-in-time
> >>>>>>> snapshot of the database from when we started streaming.
> >>>>>>>
> >>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> >>>>>>> currently support efficient offsets for key selectors
> >>>>>>> (
> >>>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> >>>> )
> >>>>>>> we implemented skip by iterating over the data. This means that a skip
> >>>>>>> of say 100000 could keep timing out the transaction without yielding
> >>>>>>> any data.
> >>>>>>>
> >>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> >>>>>>> obvious place to insert a last metadata row.
> >>>>>>>
> >>>>>>> [4] For example they have a constraint that documents "a" and "z"
> >>>>>>> cannot both be in the database at the same time. But when iterating
> >>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
> >>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
> >>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
> >>>>>>> "relaxed" constrains:
> >>>>>>>
> >>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
This is exactly what I did in my experiment.
> I don't think a whole new API is required here
Some existing endpoints return lists and not objects. Which means we have no place to return bookmark.
On 2020/04/23 21:33:49, Glynn Bird <gl...@gmail.com> wrote:
> I don't think a whole new API is required here, but I would like to see
> some sort of "bookmark" facility for _all_docs and views, as pagination
> with the current API is awkward.
>
> I would imagine it working as follows:
>
> // first request
> curl $URL/mydb/_all_docs?startkey="aardvark"&endkey="moose"&limit=10
>
> ^ the user has specified the range of values that they want, and limit
> defines the "page size", if you will.
>
> If the response to the first request contained a bookmark, the
> users's second and subsequent requests could look like:
>
> curl $URL/mydb/_all_docs?bookmark=qfqwfqwfqfqw // "get me page 2 of the
> result set"
> curl $URL/mydb/_all_docs?bookmark=iihwhfwfwhwh // "get me page 3 of the
> result set"
>
> So the bookmark need only encode:
>
> - limit (the page size)
> - starkey - that is the previous page's endkey "+1" e.g dog/u0000
> - endkey - the queries upper bound, as defined in the first request
>
> Obviously there's other parameters to think through (start_key, end_key,
> start_key_docid, descending etc) but this small iteration of the API
> doesn't add any more clutter to request path while making it substantially
> easier for the end user or client libraries to iterate through a result set.
>
> On Thu, 23 Apr 2020 at 22:14, Paul Davis <pa...@gmail.com>
> wrote:
>
> > I'd agree that my initial reaction to cursor was that its not a great
> > fit, but there does seem to be the existing name used in the greater
> > REST world for this sort of pagination so I'm not concerned about
> > using that terminology.
> >
> > I'm generally on board with allowing and setting some default sane
> > limits on pages. We probably should have done that quite awhile ago
> > after moving to native clustering and now that we have FDB limits I
> > think it makes even more sense to have an API that does not lend
> > itself to crazy errors when people are just trying to poke at an API.
> >
> > I think we're all on board that one of the goals is to make sure that
> > clients don't accidentally misinterpret a response. That is, we're
> > trying to be quite diligent that a user doesn't get 1000 rows and not
> > realize there's another 10 that were beyond the limit. The bookmark
> > approach with hard caps seems like a generally fine approach to me.
> > The current approach users extra URL path segments to try and avoid
> > this confusion. I wonder if we should consider starting to properly
> > version our API using one of the many schemes that are used. Having
> > read through a few articles I don't have a very clear favorite though.
> >
> > As to this particular proposal I do see a couple issues:
> >
> > `total` - We can do this in most cases fairly easily. Though it's a
> > bit odd for continuous changes.
> >
> > `complete` - I'm not sure whether this is entirely possible given the
> > API that FDB presents us. Specifically, when we set a range and we get
> > back exactly $num_rows in the response, if the data set ended at
> > exactly that page I don't think the `more` flag from fdb would tell us
> > that. So we'd have a clunky UX there where we say not complete but the
> > next page is empty. That's also not to mention that depending on
> > whether we're looking at snapshots and so on that there's no way for
> > us to know between stateless requests whether there were more rows
> > added to the end.
> >
> > `page` - This one is just hard/impossible to calculate. FDB doesn't
> > provide us with offsets or even an efficient "about how many rows in
> > this range?" type queries so providing that would be both inaccurate
> > and fairly difficult/expensive to calculate. In some cases I think we
> > could have something maybe close that didn't suck too badly, but it'd
> > also fall down for changes as well due to the way that updates reorder
> > them.
> >
> > `update_seq` - I'm just not sure on when this would be useful or what
> > it would refer to. Maybe a version stamp of the last change for that
> > request? If we had a future API that asked for a snapshot access then
> > maybe? But if we did do something there with versionstamps or read
> > versions I'd expect that to come with the rest of the API.
> >
> > For the bookmark fields:
> >
> > `direction` vs `descending` seems like a field duplication to me.
> >
> > `page` - This would seem to suggest we could skip to a certain
> > location in the results numerically which we are not able to do with
> > the FDB API.
> >
> > `last_key` vs `start_key` seems like a field duplication. We don't
> > need to know where things started I don't think. Just where to start
> > from and where to end.
> >
> > `update_seq` - is same as earlier. Not entirely sure on the intent there.
> >
> > `timestamp` - Expiring bookmarks based on time does not seem like a
> > good idea. Both for clock skew and why bother when this would
> > functionally just be a convenience API that users could already
> > implement for themselves.
> >
> > Another thing might also be to provide our bookmark as a full link
> > that seems to be fairly standard REST practice these days. Something
> > that clients don't have to do any logic with so that we're free to
> > change the implementation.
> >
> > And lastly, I don't think we should be neglecting the _changes API as
> > part of this discussion. I realize that we'll need to support the
> > older streaming semantics if we want to maintain replication
> > compatibility (which I think we'll all agree is a Good Thing) but it
> > also feels a bit wrong to ignore it as part of this work if we're
> > going to be modernizing our APIs. Though if we do pick up a good
> > versioning scheme then we could theoretically make those changes
> > easily enough. Plus, who doesn't want to rewrite chttpd to be a whole
> > lot less... chttpd-y?
> >
> >
> > On Thu, Apr 23, 2020 at 1:43 PM Robert Samuel Newson <rn...@apache.org>
> > wrote:
> > >
> > >
> > > I think it's a key difference from "cursor" as I've seen them elsewhere,
> > that ours will point at an ever changing database, you couldn't seamlessly
> > cursor through a large data set, one "page" at a time.
> > >
> > > Bookmarks began in search (raises guilty hand) in order to address a
> > Lucene-specific issue (that high values of "skip" are incredibly
> > inefficient, using lots of RAM). That is not true for CouchDB's own
> > indexes, which can be navigated perfectly with
> > startkey/endkey/startkey_docid/endkey_docid, etc.
> > >
> > > I guess I'm not helping much with these observations but I wouldn't like
> > to see CouchDB gain an additional and ugly method of doing something
> > already possible.
> > >
> > > B.
> > >
> > > > On 23 Apr 2020, at 19:02, Joan Touzet <wo...@apache.org> wrote:
> > > >
> > > > I realise this is bikeshedding, but I guess that's kind of the
> > point... Everything below is my opinion, not "fact."
> > > >
> > > > It's unfortunate we need a new endpoint for all of this. In a vacuum I
> > might have just suggested we use the semantics we already have, perhaps
> > with ?from= instead of ?since= .
> > > >
> > > > "page" only works if the size of a page is well known, either by
> > server preference or directly in the URL. If I ask for:
> > > >
> > > > GET /{db}/_all_docs?limit=20&page=3
> > > >
> > > > I know that I'm always going to get document 41 through 60 in the
> > default collation order.
> > > >
> > > > There's a *fantastic* summary of examples from popular REST APIs here:
> > > >
> > > >
> > https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
> > > >
> > > > We are *pretty close* to what a cursor means in those other examples,
> > except for the fact that our cursor can go stale/invalid after a short time.
> > > >
> > > > Bob, could you be a bit more detailed in your explanation how our
> > definition isn't close to these? Or did you mean SQL CURSOR (which is
> > something entirely different?) If so, I'm fine with this being a REST API
> > cursor - something clearly distinct.
> > > >
> > > > I come back to wanting to preserve the existing endpoint syntax and
> > naming, without new endpoints, but specifying this new FDB token via
> > ?cursor= and this being the trigger for the new behaviour. At some point,
> > we simply stop accepting ?since= tokens. This seems inline with other
> > popular REST APIs.
> > > >
> > > > -Joan "still sick and not sleeping right" Touzet
> > > >
> > > >
> > > > On 2020-04-23 12:30, Robert Newson wrote:
> > > >> cursor has established meaning in other databases and ours would not
> > be very close to them. I don’t think it’s a good idea.
> > > >> B.
> > > >>> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
> > > >>>
> > > >>>
> > > >>>>
> > > >>>> The best I could come up with is replacing page with
> > > >>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> > > >>> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
> > > >>>
> > > >>>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
> > > >>>> I agree with Bob that page doesn't make sense as an endpoint. I'm
> > also
> > > >>>> rubbish with naming. The best I could come up with is replacing
> > page with
> > > >>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> > > >>>> All the fields in the bookmark make sense except timestamp. Why
> > would it
> > > >>>> matter if the timestamp is old? What happens if a node's time is an
> > hour
> > > >>>> behind another node?
> > > >>>>
> > > >>>>
> > > >>>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org>
> > wrote:
> > > >>>>>
> > > >>>>> - page is to provide some notion of progress for user
> > > >>>>> - timestamp - I was thinking that we should drop requests if user
> > would
> > > >>>>> try to pass bookmark created an hour ago.
> > > >>>>>
> > > >>>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org>
> > wrote:
> > > >>>>>> "page" and "page number" are odd to me as these don't exist as
> > concepts,
> > > >>>>> I'd rather not invent them. I note there's no mention of page
> > size, which
> > > >>>>> makes "page number" very vague.
> > > >>>>>>
> > > >>>>>> What is "timestamp" in the bookmark and what effect does it have
> > when
> > > >>>>> the bookmark is passed back in?
> > > >>>>>>
> > > >>>>>> I guess, why does the bookmark include so much extraneous data?
> > Items
> > > >>>>> that are not needed to find the fdb key to begin the next response
> > from.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org>
> > wrote:
> > > >>>>>>>
> > > >>>>>>> Hello everyone,
> > > >>>>>>>
> > > >>>>>>> Based on the discussions on the thread I would like to propose a
> > > >>>>> number of first steps:
> > > >>>>>>> 1) introduce new endpoints
> > > >>>>>>> - {db}/_all_docs/page
> > > >>>>>>> - {db}/_all_docs/queries/page
> > > >>>>>>> - _all_dbs/page
> > > >>>>>>> - _dbs_info/page
> > > >>>>>>> - {db}/_design/{ddoc}/_view/{view}/page
> > > >>>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > >>>>>>> - {db}/_find/page
> > > >>>>>>>
> > > >>>>>>> These new endpoints would act as follows:
> > > >>>>>>> - don't use delayed responses
> > > >>>>>>> - return object with following structure
> > > >>>>>>> ```
> > > >>>>>>> {
> > > >>>>>>> "total": Total,
> > > >>>>>>> "bookmark": base64 encoded opaque value,
> > > >>>>>>> "completed": true | false,
> > > >>>>>>> "update_seq": when available,
> > > >>>>>>> "page": current page number,
> > > >>>>>>> "items": [
> > > >>>>>>> ]
> > > >>>>>>> }
> > > >>>>>>> ```
> > > >>>>>>> - the bookmark would include following data (base64 or
> > protobuff???):
> > > >>>>>>> - direction
> > > >>>>>>> - page
> > > >>>>>>> - descending
> > > >>>>>>> - endkey
> > > >>>>>>> - endkey_docid
> > > >>>>>>> - inclusive_end
> > > >>>>>>> - startkey
> > > >>>>>>> - startkey_docid
> > > >>>>>>> - last_key
> > > >>>>>>> - update_seq
> > > >>>>>>> - timestamp
> > > >>>>>>> ```
> > > >>>>>>>
> > > >>>>>>> 2) Implement per-endpoint configurable max limits
> > > >>>>>>> ```
> > > >>>>>>> _all_docs = 5000
> > > >>>>>>> _all_docs/queries = 5000
> > > >>>>>>> _all_dbs = 5000
> > > >>>>>>> _dbs_info = 5000
> > > >>>>>>> _view = 2500
> > > >>>>>>> _view/queries = 2500
> > > >>>>>>> _find = 2500
> > > >>>>>>> ```
> > > >>>>>>>
> > > >>>>>>> Latter (after few years) CouchDB would deprecate and remove old
> > > >>>>> endpoints.
> > > >>>>>>>
> > > >>>>>>> Best regards,
> > > >>>>>>> iilyak
> > > >>>>>>>
> > > >>>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org>
> > wrote:
> > > >>>>>>>> Hello everyone,
> > > >>>>>>>>
> > > >>>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
> > > >>>>> CouchDB 4.x
> > > >>>>>>>>
> > > >>>>>>>> By "streaming APIs" I mean APIs which stream data in row as it
> > gets
> > > >>>>>>>> read from the database. These are the endpoints I was thinking
> > of:
> > > >>>>>>>>
> > > >>>>>>>> _all_docs, _all_dbs, _dbs_info and query results
> > > >>>>>>>>
> > > >>>>>>>> I want to focus on what happens when FoundationDB transactions
> > > >>>>>>>> time-out after 5 seconds. Currently, all those APIs except
> > _changes[1]
> > > >>>>>>>> feeds, will crash or freeze. The reason is because the
> > > >>>>>>>> transaction_too_old error at the end of 5 seconds is retry-able
> > by
> > > >>>>>>>> default, so the request handlers run again and end up shoving
> > the
> > > >>>>>>>> whole request down the socket again, headers and all, which is
> > > >>>>>>>> obviously broken and not what we want.
> > > >>>>>>>>
> > > >>>>>>>> There are few alternatives discussed in couchdb-dev channel.
> > I'll
> > > >>>>>>>> present some behaviors but feel free to add more. Some ideas
> > might
> > > >>>>>>>> have been discounted on the IRC discussion already but I'll
> > present
> > > >>>>>>>> them anyway in case is sparks further conversation:
> > > >>>>>>>>
> > > >>>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and
> > continue
> > > >>>>>>>> streaming the data from the next key after last emitted in the
> > > >>>>>>>> previous transaction. Document the API behavior change that it
> > may
> > > >>>>>>>> present a view of the data is never a point-in-time[4] snapshot
> > of the
> > > >>>>>>>> DB.
> > > >>>>>>>>
> > > >>>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > >>>>>>>> don't have to change to continue using these CouchDB 4.0
> > endpoints
> > > >>>>>>>> - This is the easiest to implement since it would re-use the
> > > >>>>>>>> implementation for _changes feed (an extra option passed to the
> > fold
> > > >>>>>>>> function).
> > > >>>>>>>> - Breaks API behavior if users relied on having a
> > point-in-time[4]
> > > >>>>>>>> snapshot view of the data.
> > > >>>>>>>>
> > > >>>>>>>> B) Simply end the stream. Let the users pass a
> > `?transaction=true`
> > > >>>>>>>> param which indicates they are aware the stream may end early
> > and so
> > > >>>>>>>> would have to paginate from the last emitted key with a skip=1.
> > This
> > > >>>>>>>> will keep the request bodies the same as current CouchDB.
> > However, if
> > > >>>>>>>> the users got all the data one request, they will end up wasting
> > > >>>>>>>> another request to see if there is more data available. If they
> > didn't
> > > >>>>>>>> get any data they might have a too large of a skip value (see
> > [2]) so
> > > >>>>>>>> would have to guess different values for start/end keys. Or
> > impose max
> > > >>>>>>>> limit for the `skip` parameter.
> > > >>>>>>>>
> > > >>>>>>>> C) End the stream and add a final metadata row like a
> > "transaction":
> > > >>>>>>>> "timeout" at the end. That will let the user know to keep
> > paginating
> > > >>>>>>>> from the last key onward. This won't work for `_all_dbs` and
> > > >>>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like
> > _changes
> > > >>>>>>>> feeds and only use this for views and and _all_docs? If we like
> > this
> > > >>>>>>>> choice, let's think what happens for those as I couldn't come
> > up with
> > > >>>>>>>> anything decent there.
> > > >>>>>>>>
> > > >>>>>>>> D) Same as C but to solve the issue with skips[2], emit a
> > bookmark
> > > >>>>>>>> "key" of where the iteration stopped and the current "skip" and
> > > >>>>>>>> "limit" params, which would keep decreasing. Then user would
> > pass
> > > >>>>>>>> those in "start_key=..." in the next request along with the
> > limit and
> > > >>>>>>>> skip params. So something like "continuation":{"skip":599,
> > "limit":5,
> > > >>>>>>>> "key":"..."}. This has the same issue with array results for
> > > >>>>>>>> `_all_dbs` and `_dbs_info`[3].
> > > >>>>>>>>
> > > >>>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum
> > values
> > > >>>>>>>> there such that response time is likely to fit in one
> > transaction.
> > > >>>>>>>> This could be tricky as different runtime environments will have
> > > >>>>>>>> different characteristics. Also, if the timeout happens there
> > isn't a
> > > >>>>>>>> a nice way to send an HTTP error since we already sent the 200
> > > >>>>>>>> response. The downside is that this might break how some users
> > use the
> > > >>>>>>>> API, if say the are using large skips and limits already.
> > Perhaps here
> > > >>>>>>>> we do both B and D, such that if users want transactional
> > behavior,
> > > >>>>>>>> they specify that `transaction=true` param and only then we
> > enforce
> > > >>>>>>>> low limit and skip maximums.
> > > >>>>>>>>
> > > >>>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
> > > >>>>>>>> snapshot view doesn't necessarily need to be tied to transaction
> > > >>>>>>>> boundaries. We could check the update sequence of the database
> > at the
> > > >>>>>>>> start of the next transaction and if it hasn't changed we can
> > continue
> > > >>>>>>>> emitting a consistent view. This can apply to C and D and would
> > just
> > > >>>>>>>> determine when the stream ends. If there are no writes
> > happening to
> > > >>>>>>>> the db, this could potential streams all the data just like
> > option A
> > > >>>>>>>> would do. Not entirely sure if this would work for views.
> > > >>>>>>>>
> > > >>>>>>>> So what do we think? I can see different combinations of
> > options here,
> > > >>>>>>>> maybe even different for each API point. For example `_all_dbs`,
> > > >>>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to
> > A but
> > > >>>>>>>> have parameters to do F, etc.
> > > >>>>>>>>
> > > >>>>>>>> Cheers,
> > > >>>>>>>> -Nick
> > > >>>>>>>>
> > > >>>>>>>> Some footnotes:
> > > >>>>>>>>
> > > >>>>>>>> [1] _changes feeds is the only one that works currently. It
> > behaves as
> > > >>>>>>>> per RFC
> > > >>>>>
> > https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> > > >>>>> .
> > > >>>>>>>> That is, we continue streaming the data by resetting the
> > transaction
> > > >>>>>>>> object and restarting from the last emitted key (db sequence in
> > this
> > > >>>>>>>> case). However, because the transaction restarts if a document
> > is
> > > >>>>>>>> updated while the streaming take place, it may appear in the
> > _changes
> > > >>>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and
> > we'd
> > > >>>>>>>> have to document it, since previously we presented this
> > point-in-time
> > > >>>>>>>> snapshot of the database from when we started streaming.
> > > >>>>>>>>
> > > >>>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB
> > doesn't
> > > >>>>>>>> currently support efficient offsets for key selectors
> > > >>>>>>>> (
> > > >>>>>
> > https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> > > >>>>> )
> > > >>>>>>>> we implemented skip by iterating over the data. This means that
> > a skip
> > > >>>>>>>> of say 100000 could keep timing out the transaction without
> > yielding
> > > >>>>>>>> any data.
> > > >>>>>>>>
> > > >>>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't
> > have an
> > > >>>>>>>> obvious place to insert a last metadata row.
> > > >>>>>>>>
> > > >>>>>>>> [4] For example they have a constraint that documents "a" and
> > "z"
> > > >>>>>>>> cannot both be in the database at the same time. But when
> > iterating
> > > >>>>>>>> it's possible that "a" was there at the start. Then by the end,
> > "a"
> > > >>>>>>>> was removed and "z" added, so both "a" and "z" would appear in
> > the
> > > >>>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit
> > the same
> > > >>>>>>>> "relaxed" constrains:
> > > >>>>>>>>
> > > >>>>>
> > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > >
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Glynn Bird <gl...@gmail.com>.
I don't think a whole new API is required here, but I would like to see
some sort of "bookmark" facility for _all_docs and views, as pagination
with the current API is awkward.
I would imagine it working as follows:
// first request
curl $URL/mydb/_all_docs?startkey="aardvark"&endkey="moose"&limit=10
^ the user has specified the range of values that they want, and limit
defines the "page size", if you will.
If the response to the first request contained a bookmark, the
users's second and subsequent requests could look like:
curl $URL/mydb/_all_docs?bookmark=qfqwfqwfqfqw // "get me page 2 of the
result set"
curl $URL/mydb/_all_docs?bookmark=iihwhfwfwhwh // "get me page 3 of the
result set"
So the bookmark need only encode:
- limit (the page size)
- starkey - that is the previous page's endkey "+1" e.g dog/u0000
- endkey - the queries upper bound, as defined in the first request
Obviously there's other parameters to think through (start_key, end_key,
start_key_docid, descending etc) but this small iteration of the API
doesn't add any more clutter to request path while making it substantially
easier for the end user or client libraries to iterate through a result set.
On Thu, 23 Apr 2020 at 22:14, Paul Davis <pa...@gmail.com>
wrote:
> I'd agree that my initial reaction to cursor was that its not a great
> fit, but there does seem to be the existing name used in the greater
> REST world for this sort of pagination so I'm not concerned about
> using that terminology.
>
> I'm generally on board with allowing and setting some default sane
> limits on pages. We probably should have done that quite awhile ago
> after moving to native clustering and now that we have FDB limits I
> think it makes even more sense to have an API that does not lend
> itself to crazy errors when people are just trying to poke at an API.
>
> I think we're all on board that one of the goals is to make sure that
> clients don't accidentally misinterpret a response. That is, we're
> trying to be quite diligent that a user doesn't get 1000 rows and not
> realize there's another 10 that were beyond the limit. The bookmark
> approach with hard caps seems like a generally fine approach to me.
> The current approach users extra URL path segments to try and avoid
> this confusion. I wonder if we should consider starting to properly
> version our API using one of the many schemes that are used. Having
> read through a few articles I don't have a very clear favorite though.
>
> As to this particular proposal I do see a couple issues:
>
> `total` - We can do this in most cases fairly easily. Though it's a
> bit odd for continuous changes.
>
> `complete` - I'm not sure whether this is entirely possible given the
> API that FDB presents us. Specifically, when we set a range and we get
> back exactly $num_rows in the response, if the data set ended at
> exactly that page I don't think the `more` flag from fdb would tell us
> that. So we'd have a clunky UX there where we say not complete but the
> next page is empty. That's also not to mention that depending on
> whether we're looking at snapshots and so on that there's no way for
> us to know between stateless requests whether there were more rows
> added to the end.
>
> `page` - This one is just hard/impossible to calculate. FDB doesn't
> provide us with offsets or even an efficient "about how many rows in
> this range?" type queries so providing that would be both inaccurate
> and fairly difficult/expensive to calculate. In some cases I think we
> could have something maybe close that didn't suck too badly, but it'd
> also fall down for changes as well due to the way that updates reorder
> them.
>
> `update_seq` - I'm just not sure on when this would be useful or what
> it would refer to. Maybe a version stamp of the last change for that
> request? If we had a future API that asked for a snapshot access then
> maybe? But if we did do something there with versionstamps or read
> versions I'd expect that to come with the rest of the API.
>
> For the bookmark fields:
>
> `direction` vs `descending` seems like a field duplication to me.
>
> `page` - This would seem to suggest we could skip to a certain
> location in the results numerically which we are not able to do with
> the FDB API.
>
> `last_key` vs `start_key` seems like a field duplication. We don't
> need to know where things started I don't think. Just where to start
> from and where to end.
>
> `update_seq` - is same as earlier. Not entirely sure on the intent there.
>
> `timestamp` - Expiring bookmarks based on time does not seem like a
> good idea. Both for clock skew and why bother when this would
> functionally just be a convenience API that users could already
> implement for themselves.
>
> Another thing might also be to provide our bookmark as a full link
> that seems to be fairly standard REST practice these days. Something
> that clients don't have to do any logic with so that we're free to
> change the implementation.
>
> And lastly, I don't think we should be neglecting the _changes API as
> part of this discussion. I realize that we'll need to support the
> older streaming semantics if we want to maintain replication
> compatibility (which I think we'll all agree is a Good Thing) but it
> also feels a bit wrong to ignore it as part of this work if we're
> going to be modernizing our APIs. Though if we do pick up a good
> versioning scheme then we could theoretically make those changes
> easily enough. Plus, who doesn't want to rewrite chttpd to be a whole
> lot less... chttpd-y?
>
>
> On Thu, Apr 23, 2020 at 1:43 PM Robert Samuel Newson <rn...@apache.org>
> wrote:
> >
> >
> > I think it's a key difference from "cursor" as I've seen them elsewhere,
> that ours will point at an ever changing database, you couldn't seamlessly
> cursor through a large data set, one "page" at a time.
> >
> > Bookmarks began in search (raises guilty hand) in order to address a
> Lucene-specific issue (that high values of "skip" are incredibly
> inefficient, using lots of RAM). That is not true for CouchDB's own
> indexes, which can be navigated perfectly with
> startkey/endkey/startkey_docid/endkey_docid, etc.
> >
> > I guess I'm not helping much with these observations but I wouldn't like
> to see CouchDB gain an additional and ugly method of doing something
> already possible.
> >
> > B.
> >
> > > On 23 Apr 2020, at 19:02, Joan Touzet <wo...@apache.org> wrote:
> > >
> > > I realise this is bikeshedding, but I guess that's kind of the
> point... Everything below is my opinion, not "fact."
> > >
> > > It's unfortunate we need a new endpoint for all of this. In a vacuum I
> might have just suggested we use the semantics we already have, perhaps
> with ?from= instead of ?since= .
> > >
> > > "page" only works if the size of a page is well known, either by
> server preference or directly in the URL. If I ask for:
> > >
> > > GET /{db}/_all_docs?limit=20&page=3
> > >
> > > I know that I'm always going to get document 41 through 60 in the
> default collation order.
> > >
> > > There's a *fantastic* summary of examples from popular REST APIs here:
> > >
> > >
> https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
> > >
> > > We are *pretty close* to what a cursor means in those other examples,
> except for the fact that our cursor can go stale/invalid after a short time.
> > >
> > > Bob, could you be a bit more detailed in your explanation how our
> definition isn't close to these? Or did you mean SQL CURSOR (which is
> something entirely different?) If so, I'm fine with this being a REST API
> cursor - something clearly distinct.
> > >
> > > I come back to wanting to preserve the existing endpoint syntax and
> naming, without new endpoints, but specifying this new FDB token via
> ?cursor= and this being the trigger for the new behaviour. At some point,
> we simply stop accepting ?since= tokens. This seems inline with other
> popular REST APIs.
> > >
> > > -Joan "still sick and not sleeping right" Touzet
> > >
> > >
> > > On 2020-04-23 12:30, Robert Newson wrote:
> > >> cursor has established meaning in other databases and ours would not
> be very close to them. I don’t think it’s a good idea.
> > >> B.
> > >>> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
> > >>>
> > >>>
> > >>>>
> > >>>> The best I could come up with is replacing page with
> > >>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> > >>> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
> > >>>
> > >>>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
> > >>>> I agree with Bob that page doesn't make sense as an endpoint. I'm
> also
> > >>>> rubbish with naming. The best I could come up with is replacing
> page with
> > >>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> > >>>> All the fields in the bookmark make sense except timestamp. Why
> would it
> > >>>> matter if the timestamp is old? What happens if a node's time is an
> hour
> > >>>> behind another node?
> > >>>>
> > >>>>
> > >>>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org>
> wrote:
> > >>>>>
> > >>>>> - page is to provide some notion of progress for user
> > >>>>> - timestamp - I was thinking that we should drop requests if user
> would
> > >>>>> try to pass bookmark created an hour ago.
> > >>>>>
> > >>>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org>
> wrote:
> > >>>>>> "page" and "page number" are odd to me as these don't exist as
> concepts,
> > >>>>> I'd rather not invent them. I note there's no mention of page
> size, which
> > >>>>> makes "page number" very vague.
> > >>>>>>
> > >>>>>> What is "timestamp" in the bookmark and what effect does it have
> when
> > >>>>> the bookmark is passed back in?
> > >>>>>>
> > >>>>>> I guess, why does the bookmark include so much extraneous data?
> Items
> > >>>>> that are not needed to find the fdb key to begin the next response
> from.
> > >>>>>>
> > >>>>>>
> > >>>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org>
> wrote:
> > >>>>>>>
> > >>>>>>> Hello everyone,
> > >>>>>>>
> > >>>>>>> Based on the discussions on the thread I would like to propose a
> > >>>>> number of first steps:
> > >>>>>>> 1) introduce new endpoints
> > >>>>>>> - {db}/_all_docs/page
> > >>>>>>> - {db}/_all_docs/queries/page
> > >>>>>>> - _all_dbs/page
> > >>>>>>> - _dbs_info/page
> > >>>>>>> - {db}/_design/{ddoc}/_view/{view}/page
> > >>>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
> > >>>>>>> - {db}/_find/page
> > >>>>>>>
> > >>>>>>> These new endpoints would act as follows:
> > >>>>>>> - don't use delayed responses
> > >>>>>>> - return object with following structure
> > >>>>>>> ```
> > >>>>>>> {
> > >>>>>>> "total": Total,
> > >>>>>>> "bookmark": base64 encoded opaque value,
> > >>>>>>> "completed": true | false,
> > >>>>>>> "update_seq": when available,
> > >>>>>>> "page": current page number,
> > >>>>>>> "items": [
> > >>>>>>> ]
> > >>>>>>> }
> > >>>>>>> ```
> > >>>>>>> - the bookmark would include following data (base64 or
> protobuff???):
> > >>>>>>> - direction
> > >>>>>>> - page
> > >>>>>>> - descending
> > >>>>>>> - endkey
> > >>>>>>> - endkey_docid
> > >>>>>>> - inclusive_end
> > >>>>>>> - startkey
> > >>>>>>> - startkey_docid
> > >>>>>>> - last_key
> > >>>>>>> - update_seq
> > >>>>>>> - timestamp
> > >>>>>>> ```
> > >>>>>>>
> > >>>>>>> 2) Implement per-endpoint configurable max limits
> > >>>>>>> ```
> > >>>>>>> _all_docs = 5000
> > >>>>>>> _all_docs/queries = 5000
> > >>>>>>> _all_dbs = 5000
> > >>>>>>> _dbs_info = 5000
> > >>>>>>> _view = 2500
> > >>>>>>> _view/queries = 2500
> > >>>>>>> _find = 2500
> > >>>>>>> ```
> > >>>>>>>
> > >>>>>>> Latter (after few years) CouchDB would deprecate and remove old
> > >>>>> endpoints.
> > >>>>>>>
> > >>>>>>> Best regards,
> > >>>>>>> iilyak
> > >>>>>>>
> > >>>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org>
> wrote:
> > >>>>>>>> Hello everyone,
> > >>>>>>>>
> > >>>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
> > >>>>> CouchDB 4.x
> > >>>>>>>>
> > >>>>>>>> By "streaming APIs" I mean APIs which stream data in row as it
> gets
> > >>>>>>>> read from the database. These are the endpoints I was thinking
> of:
> > >>>>>>>>
> > >>>>>>>> _all_docs, _all_dbs, _dbs_info and query results
> > >>>>>>>>
> > >>>>>>>> I want to focus on what happens when FoundationDB transactions
> > >>>>>>>> time-out after 5 seconds. Currently, all those APIs except
> _changes[1]
> > >>>>>>>> feeds, will crash or freeze. The reason is because the
> > >>>>>>>> transaction_too_old error at the end of 5 seconds is retry-able
> by
> > >>>>>>>> default, so the request handlers run again and end up shoving
> the
> > >>>>>>>> whole request down the socket again, headers and all, which is
> > >>>>>>>> obviously broken and not what we want.
> > >>>>>>>>
> > >>>>>>>> There are few alternatives discussed in couchdb-dev channel.
> I'll
> > >>>>>>>> present some behaviors but feel free to add more. Some ideas
> might
> > >>>>>>>> have been discounted on the IRC discussion already but I'll
> present
> > >>>>>>>> them anyway in case is sparks further conversation:
> > >>>>>>>>
> > >>>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and
> continue
> > >>>>>>>> streaming the data from the next key after last emitted in the
> > >>>>>>>> previous transaction. Document the API behavior change that it
> may
> > >>>>>>>> present a view of the data is never a point-in-time[4] snapshot
> of the
> > >>>>>>>> DB.
> > >>>>>>>>
> > >>>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > >>>>>>>> don't have to change to continue using these CouchDB 4.0
> endpoints
> > >>>>>>>> - This is the easiest to implement since it would re-use the
> > >>>>>>>> implementation for _changes feed (an extra option passed to the
> fold
> > >>>>>>>> function).
> > >>>>>>>> - Breaks API behavior if users relied on having a
> point-in-time[4]
> > >>>>>>>> snapshot view of the data.
> > >>>>>>>>
> > >>>>>>>> B) Simply end the stream. Let the users pass a
> `?transaction=true`
> > >>>>>>>> param which indicates they are aware the stream may end early
> and so
> > >>>>>>>> would have to paginate from the last emitted key with a skip=1.
> This
> > >>>>>>>> will keep the request bodies the same as current CouchDB.
> However, if
> > >>>>>>>> the users got all the data one request, they will end up wasting
> > >>>>>>>> another request to see if there is more data available. If they
> didn't
> > >>>>>>>> get any data they might have a too large of a skip value (see
> [2]) so
> > >>>>>>>> would have to guess different values for start/end keys. Or
> impose max
> > >>>>>>>> limit for the `skip` parameter.
> > >>>>>>>>
> > >>>>>>>> C) End the stream and add a final metadata row like a
> "transaction":
> > >>>>>>>> "timeout" at the end. That will let the user know to keep
> paginating
> > >>>>>>>> from the last key onward. This won't work for `_all_dbs` and
> > >>>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like
> _changes
> > >>>>>>>> feeds and only use this for views and and _all_docs? If we like
> this
> > >>>>>>>> choice, let's think what happens for those as I couldn't come
> up with
> > >>>>>>>> anything decent there.
> > >>>>>>>>
> > >>>>>>>> D) Same as C but to solve the issue with skips[2], emit a
> bookmark
> > >>>>>>>> "key" of where the iteration stopped and the current "skip" and
> > >>>>>>>> "limit" params, which would keep decreasing. Then user would
> pass
> > >>>>>>>> those in "start_key=..." in the next request along with the
> limit and
> > >>>>>>>> skip params. So something like "continuation":{"skip":599,
> "limit":5,
> > >>>>>>>> "key":"..."}. This has the same issue with array results for
> > >>>>>>>> `_all_dbs` and `_dbs_info`[3].
> > >>>>>>>>
> > >>>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum
> values
> > >>>>>>>> there such that response time is likely to fit in one
> transaction.
> > >>>>>>>> This could be tricky as different runtime environments will have
> > >>>>>>>> different characteristics. Also, if the timeout happens there
> isn't a
> > >>>>>>>> a nice way to send an HTTP error since we already sent the 200
> > >>>>>>>> response. The downside is that this might break how some users
> use the
> > >>>>>>>> API, if say the are using large skips and limits already.
> Perhaps here
> > >>>>>>>> we do both B and D, such that if users want transactional
> behavior,
> > >>>>>>>> they specify that `transaction=true` param and only then we
> enforce
> > >>>>>>>> low limit and skip maximums.
> > >>>>>>>>
> > >>>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
> > >>>>>>>> snapshot view doesn't necessarily need to be tied to transaction
> > >>>>>>>> boundaries. We could check the update sequence of the database
> at the
> > >>>>>>>> start of the next transaction and if it hasn't changed we can
> continue
> > >>>>>>>> emitting a consistent view. This can apply to C and D and would
> just
> > >>>>>>>> determine when the stream ends. If there are no writes
> happening to
> > >>>>>>>> the db, this could potential streams all the data just like
> option A
> > >>>>>>>> would do. Not entirely sure if this would work for views.
> > >>>>>>>>
> > >>>>>>>> So what do we think? I can see different combinations of
> options here,
> > >>>>>>>> maybe even different for each API point. For example `_all_dbs`,
> > >>>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to
> A but
> > >>>>>>>> have parameters to do F, etc.
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>> -Nick
> > >>>>>>>>
> > >>>>>>>> Some footnotes:
> > >>>>>>>>
> > >>>>>>>> [1] _changes feeds is the only one that works currently. It
> behaves as
> > >>>>>>>> per RFC
> > >>>>>
> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> > >>>>> .
> > >>>>>>>> That is, we continue streaming the data by resetting the
> transaction
> > >>>>>>>> object and restarting from the last emitted key (db sequence in
> this
> > >>>>>>>> case). However, because the transaction restarts if a document
> is
> > >>>>>>>> updated while the streaming take place, it may appear in the
> _changes
> > >>>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and
> we'd
> > >>>>>>>> have to document it, since previously we presented this
> point-in-time
> > >>>>>>>> snapshot of the database from when we started streaming.
> > >>>>>>>>
> > >>>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB
> doesn't
> > >>>>>>>> currently support efficient offsets for key selectors
> > >>>>>>>> (
> > >>>>>
> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> > >>>>> )
> > >>>>>>>> we implemented skip by iterating over the data. This means that
> a skip
> > >>>>>>>> of say 100000 could keep timing out the transaction without
> yielding
> > >>>>>>>> any data.
> > >>>>>>>>
> > >>>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't
> have an
> > >>>>>>>> obvious place to insert a last metadata row.
> > >>>>>>>>
> > >>>>>>>> [4] For example they have a constraint that documents "a" and
> "z"
> > >>>>>>>> cannot both be in the database at the same time. But when
> iterating
> > >>>>>>>> it's possible that "a" was there at the start. Then by the end,
> "a"
> > >>>>>>>> was removed and "z" added, so both "a" and "z" would appear in
> the
> > >>>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit
> the same
> > >>>>>>>> "relaxed" constrains:
> > >>>>>>>>
> > >>>>>
> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
> On versioning, I've not seen a better article than this one: https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/
I wouldn't propose new endpoint if we would have a strong story for API versioning. Currently we don't.
BTW we could put these new endpoints into a new namespace for example `_v2/_all_docs`. In this case we wouldn't need to invent new names.
Best regards,
iilyak
On 2020/04/23 21:31:41, Robert Samuel Newson <rn...@apache.org> wrote:
> On versioning, I've not seen a better article than this one: https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/
>
> For _changes, definitely agree we should be including it in this discussion, it is the only endpoint with, in theory, an eternal response, and I think that's a bug not a feature these days. CouchDB exists in a wider ecosystem (and often behind a load balancer), it would be good to define an upper bound on how long you can listen before being forced to query again.
>
> B.
>
> > On 23 Apr 2020, at 22:15, Paul Davis <pa...@gmail.com> wrote:
> >
> > I'd agree that my initial reaction to cursor was that its not a great
> > fit, but there does seem to be the existing name used in the greater
> > REST world for this sort of pagination so I'm not concerned about
> > using that terminology.
> >
> > I'm generally on board with allowing and setting some default sane
> > limits on pages. We probably should have done that quite awhile ago
> > after moving to native clustering and now that we have FDB limits I
> > think it makes even more sense to have an API that does not lend
> > itself to crazy errors when people are just trying to poke at an API.
> >
> > I think we're all on board that one of the goals is to make sure that
> > clients don't accidentally misinterpret a response. That is, we're
> > trying to be quite diligent that a user doesn't get 1000 rows and not
> > realize there's another 10 that were beyond the limit. The bookmark
> > approach with hard caps seems like a generally fine approach to me.
> > The current approach users extra URL path segments to try and avoid
> > this confusion. I wonder if we should consider starting to properly
> > version our API using one of the many schemes that are used. Having
> > read through a few articles I don't have a very clear favorite though.
> >
> > As to this particular proposal I do see a couple issues:
> >
> > `total` - We can do this in most cases fairly easily. Though it's a
> > bit odd for continuous changes.
> >
> > `complete` - I'm not sure whether this is entirely possible given the
> > API that FDB presents us. Specifically, when we set a range and we get
> > back exactly $num_rows in the response, if the data set ended at
> > exactly that page I don't think the `more` flag from fdb would tell us
> > that. So we'd have a clunky UX there where we say not complete but the
> > next page is empty. That's also not to mention that depending on
> > whether we're looking at snapshots and so on that there's no way for
> > us to know between stateless requests whether there were more rows
> > added to the end.
> >
> > `page` - This one is just hard/impossible to calculate. FDB doesn't
> > provide us with offsets or even an efficient "about how many rows in
> > this range?" type queries so providing that would be both inaccurate
> > and fairly difficult/expensive to calculate. In some cases I think we
> > could have something maybe close that didn't suck too badly, but it'd
> > also fall down for changes as well due to the way that updates reorder
> > them.
> >
> > `update_seq` - I'm just not sure on when this would be useful or what
> > it would refer to. Maybe a version stamp of the last change for that
> > request? If we had a future API that asked for a snapshot access then
> > maybe? But if we did do something there with versionstamps or read
> > versions I'd expect that to come with the rest of the API.
> >
> > For the bookmark fields:
> >
> > `direction` vs `descending` seems like a field duplication to me.
> >
> > `page` - This would seem to suggest we could skip to a certain
> > location in the results numerically which we are not able to do with
> > the FDB API.
> >
> > `last_key` vs `start_key` seems like a field duplication. We don't
> > need to know where things started I don't think. Just where to start
> > from and where to end.
> >
> > `update_seq` - is same as earlier. Not entirely sure on the intent there.
> >
> > `timestamp` - Expiring bookmarks based on time does not seem like a
> > good idea. Both for clock skew and why bother when this would
> > functionally just be a convenience API that users could already
> > implement for themselves.
> >
> > Another thing might also be to provide our bookmark as a full link
> > that seems to be fairly standard REST practice these days. Something
> > that clients don't have to do any logic with so that we're free to
> > change the implementation.
> >
> > And lastly, I don't think we should be neglecting the _changes API as
> > part of this discussion. I realize that we'll need to support the
> > older streaming semantics if we want to maintain replication
> > compatibility (which I think we'll all agree is a Good Thing) but it
> > also feels a bit wrong to ignore it as part of this work if we're
> > going to be modernizing our APIs. Though if we do pick up a good
> > versioning scheme then we could theoretically make those changes
> > easily enough. Plus, who doesn't want to rewrite chttpd to be a whole
> > lot less... chttpd-y?
> >
> >
> > On Thu, Apr 23, 2020 at 1:43 PM Robert Samuel Newson <rn...@apache.org> wrote:
> >>
> >>
> >> I think it's a key difference from "cursor" as I've seen them elsewhere, that ours will point at an ever changing database, you couldn't seamlessly cursor through a large data set, one "page" at a time.
> >>
> >> Bookmarks began in search (raises guilty hand) in order to address a Lucene-specific issue (that high values of "skip" are incredibly inefficient, using lots of RAM). That is not true for CouchDB's own indexes, which can be navigated perfectly with startkey/endkey/startkey_docid/endkey_docid, etc.
> >>
> >> I guess I'm not helping much with these observations but I wouldn't like to see CouchDB gain an additional and ugly method of doing something already possible.
> >>
> >> B.
> >>
> >>> On 23 Apr 2020, at 19:02, Joan Touzet <wo...@apache.org> wrote:
> >>>
> >>> I realise this is bikeshedding, but I guess that's kind of the point... Everything below is my opinion, not "fact."
> >>>
> >>> It's unfortunate we need a new endpoint for all of this. In a vacuum I might have just suggested we use the semantics we already have, perhaps with ?from= instead of ?since= .
> >>>
> >>> "page" only works if the size of a page is well known, either by server preference or directly in the URL. If I ask for:
> >>>
> >>> GET /{db}/_all_docs?limit=20&page=3
> >>>
> >>> I know that I'm always going to get document 41 through 60 in the default collation order.
> >>>
> >>> There's a *fantastic* summary of examples from popular REST APIs here:
> >>>
> >>> https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
> >>>
> >>> We are *pretty close* to what a cursor means in those other examples, except for the fact that our cursor can go stale/invalid after a short time.
> >>>
> >>> Bob, could you be a bit more detailed in your explanation how our definition isn't close to these? Or did you mean SQL CURSOR (which is something entirely different?) If so, I'm fine with this being a REST API cursor - something clearly distinct.
> >>>
> >>> I come back to wanting to preserve the existing endpoint syntax and naming, without new endpoints, but specifying this new FDB token via ?cursor= and this being the trigger for the new behaviour. At some point, we simply stop accepting ?since= tokens. This seems inline with other popular REST APIs.
> >>>
> >>> -Joan "still sick and not sleeping right" Touzet
> >>>
> >>>
> >>> On 2020-04-23 12:30, Robert Newson wrote:
> >>>> cursor has established meaning in other databases and ours would not be very close to them. I don’t think it’s a good idea.
> >>>> B.
> >>>>> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> The best I could come up with is replacing page with
> >>>>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> >>>>> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
> >>>>>
> >>>>>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
> >>>>>> I agree with Bob that page doesn't make sense as an endpoint. I'm also
> >>>>>> rubbish with naming. The best I could come up with is replacing page with
> >>>>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> >>>>>> All the fields in the bookmark make sense except timestamp. Why would it
> >>>>>> matter if the timestamp is old? What happens if a node's time is an hour
> >>>>>> behind another node?
> >>>>>>
> >>>>>>
> >>>>>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
> >>>>>>>
> >>>>>>> - page is to provide some notion of progress for user
> >>>>>>> - timestamp - I was thinking that we should drop requests if user would
> >>>>>>> try to pass bookmark created an hour ago.
> >>>>>>>
> >>>>>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
> >>>>>>>> "page" and "page number" are odd to me as these don't exist as concepts,
> >>>>>>> I'd rather not invent them. I note there's no mention of page size, which
> >>>>>>> makes "page number" very vague.
> >>>>>>>>
> >>>>>>>> What is "timestamp" in the bookmark and what effect does it have when
> >>>>>>> the bookmark is passed back in?
> >>>>>>>>
> >>>>>>>> I guess, why does the bookmark include so much extraneous data? Items
> >>>>>>> that are not needed to find the fdb key to begin the next response from.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
> >>>>>>>>>
> >>>>>>>>> Hello everyone,
> >>>>>>>>>
> >>>>>>>>> Based on the discussions on the thread I would like to propose a
> >>>>>>> number of first steps:
> >>>>>>>>> 1) introduce new endpoints
> >>>>>>>>> - {db}/_all_docs/page
> >>>>>>>>> - {db}/_all_docs/queries/page
> >>>>>>>>> - _all_dbs/page
> >>>>>>>>> - _dbs_info/page
> >>>>>>>>> - {db}/_design/{ddoc}/_view/{view}/page
> >>>>>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
> >>>>>>>>> - {db}/_find/page
> >>>>>>>>>
> >>>>>>>>> These new endpoints would act as follows:
> >>>>>>>>> - don't use delayed responses
> >>>>>>>>> - return object with following structure
> >>>>>>>>> ```
> >>>>>>>>> {
> >>>>>>>>> "total": Total,
> >>>>>>>>> "bookmark": base64 encoded opaque value,
> >>>>>>>>> "completed": true | false,
> >>>>>>>>> "update_seq": when available,
> >>>>>>>>> "page": current page number,
> >>>>>>>>> "items": [
> >>>>>>>>> ]
> >>>>>>>>> }
> >>>>>>>>> ```
> >>>>>>>>> - the bookmark would include following data (base64 or protobuff???):
> >>>>>>>>> - direction
> >>>>>>>>> - page
> >>>>>>>>> - descending
> >>>>>>>>> - endkey
> >>>>>>>>> - endkey_docid
> >>>>>>>>> - inclusive_end
> >>>>>>>>> - startkey
> >>>>>>>>> - startkey_docid
> >>>>>>>>> - last_key
> >>>>>>>>> - update_seq
> >>>>>>>>> - timestamp
> >>>>>>>>> ```
> >>>>>>>>>
> >>>>>>>>> 2) Implement per-endpoint configurable max limits
> >>>>>>>>> ```
> >>>>>>>>> _all_docs = 5000
> >>>>>>>>> _all_docs/queries = 5000
> >>>>>>>>> _all_dbs = 5000
> >>>>>>>>> _dbs_info = 5000
> >>>>>>>>> _view = 2500
> >>>>>>>>> _view/queries = 2500
> >>>>>>>>> _find = 2500
> >>>>>>>>> ```
> >>>>>>>>>
> >>>>>>>>> Latter (after few years) CouchDB would deprecate and remove old
> >>>>>>> endpoints.
> >>>>>>>>>
> >>>>>>>>> Best regards,
> >>>>>>>>> iilyak
> >>>>>>>>>
> >>>>>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> >>>>>>>>>> Hello everyone,
> >>>>>>>>>>
> >>>>>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
> >>>>>>> CouchDB 4.x
> >>>>>>>>>>
> >>>>>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
> >>>>>>>>>> read from the database. These are the endpoints I was thinking of:
> >>>>>>>>>>
> >>>>>>>>>> _all_docs, _all_dbs, _dbs_info and query results
> >>>>>>>>>>
> >>>>>>>>>> I want to focus on what happens when FoundationDB transactions
> >>>>>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> >>>>>>>>>> feeds, will crash or freeze. The reason is because the
> >>>>>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
> >>>>>>>>>> default, so the request handlers run again and end up shoving the
> >>>>>>>>>> whole request down the socket again, headers and all, which is
> >>>>>>>>>> obviously broken and not what we want.
> >>>>>>>>>>
> >>>>>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
> >>>>>>>>>> present some behaviors but feel free to add more. Some ideas might
> >>>>>>>>>> have been discounted on the IRC discussion already but I'll present
> >>>>>>>>>> them anyway in case is sparks further conversation:
> >>>>>>>>>>
> >>>>>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
> >>>>>>>>>> streaming the data from the next key after last emitted in the
> >>>>>>>>>> previous transaction. Document the API behavior change that it may
> >>>>>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
> >>>>>>>>>> DB.
> >>>>>>>>>>
> >>>>>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> >>>>>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
> >>>>>>>>>> - This is the easiest to implement since it would re-use the
> >>>>>>>>>> implementation for _changes feed (an extra option passed to the fold
> >>>>>>>>>> function).
> >>>>>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
> >>>>>>>>>> snapshot view of the data.
> >>>>>>>>>>
> >>>>>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
> >>>>>>>>>> param which indicates they are aware the stream may end early and so
> >>>>>>>>>> would have to paginate from the last emitted key with a skip=1. This
> >>>>>>>>>> will keep the request bodies the same as current CouchDB. However, if
> >>>>>>>>>> the users got all the data one request, they will end up wasting
> >>>>>>>>>> another request to see if there is more data available. If they didn't
> >>>>>>>>>> get any data they might have a too large of a skip value (see [2]) so
> >>>>>>>>>> would have to guess different values for start/end keys. Or impose max
> >>>>>>>>>> limit for the `skip` parameter.
> >>>>>>>>>>
> >>>>>>>>>> C) End the stream and add a final metadata row like a "transaction":
> >>>>>>>>>> "timeout" at the end. That will let the user know to keep paginating
> >>>>>>>>>> from the last key onward. This won't work for `_all_dbs` and
> >>>>>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> >>>>>>>>>> feeds and only use this for views and and _all_docs? If we like this
> >>>>>>>>>> choice, let's think what happens for those as I couldn't come up with
> >>>>>>>>>> anything decent there.
> >>>>>>>>>>
> >>>>>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
> >>>>>>>>>> "key" of where the iteration stopped and the current "skip" and
> >>>>>>>>>> "limit" params, which would keep decreasing. Then user would pass
> >>>>>>>>>> those in "start_key=..." in the next request along with the limit and
> >>>>>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
> >>>>>>>>>> "key":"..."}. This has the same issue with array results for
> >>>>>>>>>> `_all_dbs` and `_dbs_info`[3].
> >>>>>>>>>>
> >>>>>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> >>>>>>>>>> there such that response time is likely to fit in one transaction.
> >>>>>>>>>> This could be tricky as different runtime environments will have
> >>>>>>>>>> different characteristics. Also, if the timeout happens there isn't a
> >>>>>>>>>> a nice way to send an HTTP error since we already sent the 200
> >>>>>>>>>> response. The downside is that this might break how some users use the
> >>>>>>>>>> API, if say the are using large skips and limits already. Perhaps here
> >>>>>>>>>> we do both B and D, such that if users want transactional behavior,
> >>>>>>>>>> they specify that `transaction=true` param and only then we enforce
> >>>>>>>>>> low limit and skip maximums.
> >>>>>>>>>>
> >>>>>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
> >>>>>>>>>> snapshot view doesn't necessarily need to be tied to transaction
> >>>>>>>>>> boundaries. We could check the update sequence of the database at the
> >>>>>>>>>> start of the next transaction and if it hasn't changed we can continue
> >>>>>>>>>> emitting a consistent view. This can apply to C and D and would just
> >>>>>>>>>> determine when the stream ends. If there are no writes happening to
> >>>>>>>>>> the db, this could potential streams all the data just like option A
> >>>>>>>>>> would do. Not entirely sure if this would work for views.
> >>>>>>>>>>
> >>>>>>>>>> So what do we think? I can see different combinations of options here,
> >>>>>>>>>> maybe even different for each API point. For example `_all_dbs`,
> >>>>>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
> >>>>>>>>>> have parameters to do F, etc.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> -Nick
> >>>>>>>>>>
> >>>>>>>>>> Some footnotes:
> >>>>>>>>>>
> >>>>>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
> >>>>>>>>>> per RFC
> >>>>>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> >>>>>>> .
> >>>>>>>>>> That is, we continue streaming the data by resetting the transaction
> >>>>>>>>>> object and restarting from the last emitted key (db sequence in this
> >>>>>>>>>> case). However, because the transaction restarts if a document is
> >>>>>>>>>> updated while the streaming take place, it may appear in the _changes
> >>>>>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> >>>>>>>>>> have to document it, since previously we presented this point-in-time
> >>>>>>>>>> snapshot of the database from when we started streaming.
> >>>>>>>>>>
> >>>>>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> >>>>>>>>>> currently support efficient offsets for key selectors
> >>>>>>>>>> (
> >>>>>>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> >>>>>>> )
> >>>>>>>>>> we implemented skip by iterating over the data. This means that a skip
> >>>>>>>>>> of say 100000 could keep timing out the transaction without yielding
> >>>>>>>>>> any data.
> >>>>>>>>>>
> >>>>>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> >>>>>>>>>> obvious place to insert a last metadata row.
> >>>>>>>>>>
> >>>>>>>>>> [4] For example they have a constraint that documents "a" and "z"
> >>>>>>>>>> cannot both be in the database at the same time. But when iterating
> >>>>>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
> >>>>>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
> >>>>>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
> >>>>>>>>>> "relaxed" constrains:
> >>>>>>>>>>
> >>>>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>
>
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Robert Samuel Newson <rn...@apache.org>.
On versioning, I've not seen a better article than this one: https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/
For _changes, definitely agree we should be including it in this discussion, it is the only endpoint with, in theory, an eternal response, and I think that's a bug not a feature these days. CouchDB exists in a wider ecosystem (and often behind a load balancer), it would be good to define an upper bound on how long you can listen before being forced to query again.
B.
> On 23 Apr 2020, at 22:15, Paul Davis <pa...@gmail.com> wrote:
>
> I'd agree that my initial reaction to cursor was that its not a great
> fit, but there does seem to be the existing name used in the greater
> REST world for this sort of pagination so I'm not concerned about
> using that terminology.
>
> I'm generally on board with allowing and setting some default sane
> limits on pages. We probably should have done that quite awhile ago
> after moving to native clustering and now that we have FDB limits I
> think it makes even more sense to have an API that does not lend
> itself to crazy errors when people are just trying to poke at an API.
>
> I think we're all on board that one of the goals is to make sure that
> clients don't accidentally misinterpret a response. That is, we're
> trying to be quite diligent that a user doesn't get 1000 rows and not
> realize there's another 10 that were beyond the limit. The bookmark
> approach with hard caps seems like a generally fine approach to me.
> The current approach users extra URL path segments to try and avoid
> this confusion. I wonder if we should consider starting to properly
> version our API using one of the many schemes that are used. Having
> read through a few articles I don't have a very clear favorite though.
>
> As to this particular proposal I do see a couple issues:
>
> `total` - We can do this in most cases fairly easily. Though it's a
> bit odd for continuous changes.
>
> `complete` - I'm not sure whether this is entirely possible given the
> API that FDB presents us. Specifically, when we set a range and we get
> back exactly $num_rows in the response, if the data set ended at
> exactly that page I don't think the `more` flag from fdb would tell us
> that. So we'd have a clunky UX there where we say not complete but the
> next page is empty. That's also not to mention that depending on
> whether we're looking at snapshots and so on that there's no way for
> us to know between stateless requests whether there were more rows
> added to the end.
>
> `page` - This one is just hard/impossible to calculate. FDB doesn't
> provide us with offsets or even an efficient "about how many rows in
> this range?" type queries so providing that would be both inaccurate
> and fairly difficult/expensive to calculate. In some cases I think we
> could have something maybe close that didn't suck too badly, but it'd
> also fall down for changes as well due to the way that updates reorder
> them.
>
> `update_seq` - I'm just not sure on when this would be useful or what
> it would refer to. Maybe a version stamp of the last change for that
> request? If we had a future API that asked for a snapshot access then
> maybe? But if we did do something there with versionstamps or read
> versions I'd expect that to come with the rest of the API.
>
> For the bookmark fields:
>
> `direction` vs `descending` seems like a field duplication to me.
>
> `page` - This would seem to suggest we could skip to a certain
> location in the results numerically which we are not able to do with
> the FDB API.
>
> `last_key` vs `start_key` seems like a field duplication. We don't
> need to know where things started I don't think. Just where to start
> from and where to end.
>
> `update_seq` - is same as earlier. Not entirely sure on the intent there.
>
> `timestamp` - Expiring bookmarks based on time does not seem like a
> good idea. Both for clock skew and why bother when this would
> functionally just be a convenience API that users could already
> implement for themselves.
>
> Another thing might also be to provide our bookmark as a full link
> that seems to be fairly standard REST practice these days. Something
> that clients don't have to do any logic with so that we're free to
> change the implementation.
>
> And lastly, I don't think we should be neglecting the _changes API as
> part of this discussion. I realize that we'll need to support the
> older streaming semantics if we want to maintain replication
> compatibility (which I think we'll all agree is a Good Thing) but it
> also feels a bit wrong to ignore it as part of this work if we're
> going to be modernizing our APIs. Though if we do pick up a good
> versioning scheme then we could theoretically make those changes
> easily enough. Plus, who doesn't want to rewrite chttpd to be a whole
> lot less... chttpd-y?
>
>
> On Thu, Apr 23, 2020 at 1:43 PM Robert Samuel Newson <rn...@apache.org> wrote:
>>
>>
>> I think it's a key difference from "cursor" as I've seen them elsewhere, that ours will point at an ever changing database, you couldn't seamlessly cursor through a large data set, one "page" at a time.
>>
>> Bookmarks began in search (raises guilty hand) in order to address a Lucene-specific issue (that high values of "skip" are incredibly inefficient, using lots of RAM). That is not true for CouchDB's own indexes, which can be navigated perfectly with startkey/endkey/startkey_docid/endkey_docid, etc.
>>
>> I guess I'm not helping much with these observations but I wouldn't like to see CouchDB gain an additional and ugly method of doing something already possible.
>>
>> B.
>>
>>> On 23 Apr 2020, at 19:02, Joan Touzet <wo...@apache.org> wrote:
>>>
>>> I realise this is bikeshedding, but I guess that's kind of the point... Everything below is my opinion, not "fact."
>>>
>>> It's unfortunate we need a new endpoint for all of this. In a vacuum I might have just suggested we use the semantics we already have, perhaps with ?from= instead of ?since= .
>>>
>>> "page" only works if the size of a page is well known, either by server preference or directly in the URL. If I ask for:
>>>
>>> GET /{db}/_all_docs?limit=20&page=3
>>>
>>> I know that I'm always going to get document 41 through 60 in the default collation order.
>>>
>>> There's a *fantastic* summary of examples from popular REST APIs here:
>>>
>>> https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
>>>
>>> We are *pretty close* to what a cursor means in those other examples, except for the fact that our cursor can go stale/invalid after a short time.
>>>
>>> Bob, could you be a bit more detailed in your explanation how our definition isn't close to these? Or did you mean SQL CURSOR (which is something entirely different?) If so, I'm fine with this being a REST API cursor - something clearly distinct.
>>>
>>> I come back to wanting to preserve the existing endpoint syntax and naming, without new endpoints, but specifying this new FDB token via ?cursor= and this being the trigger for the new behaviour. At some point, we simply stop accepting ?since= tokens. This seems inline with other popular REST APIs.
>>>
>>> -Joan "still sick and not sleeping right" Touzet
>>>
>>>
>>> On 2020-04-23 12:30, Robert Newson wrote:
>>>> cursor has established meaning in other databases and ours would not be very close to them. I don’t think it’s a good idea.
>>>> B.
>>>>> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> The best I could come up with is replacing page with
>>>>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
>>>>> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
>>>>>
>>>>>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
>>>>>> I agree with Bob that page doesn't make sense as an endpoint. I'm also
>>>>>> rubbish with naming. The best I could come up with is replacing page with
>>>>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
>>>>>> All the fields in the bookmark make sense except timestamp. Why would it
>>>>>> matter if the timestamp is old? What happens if a node's time is an hour
>>>>>> behind another node?
>>>>>>
>>>>>>
>>>>>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
>>>>>>>
>>>>>>> - page is to provide some notion of progress for user
>>>>>>> - timestamp - I was thinking that we should drop requests if user would
>>>>>>> try to pass bookmark created an hour ago.
>>>>>>>
>>>>>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>>>>> "page" and "page number" are odd to me as these don't exist as concepts,
>>>>>>> I'd rather not invent them. I note there's no mention of page size, which
>>>>>>> makes "page number" very vague.
>>>>>>>>
>>>>>>>> What is "timestamp" in the bookmark and what effect does it have when
>>>>>>> the bookmark is passed back in?
>>>>>>>>
>>>>>>>> I guess, why does the bookmark include so much extraneous data? Items
>>>>>>> that are not needed to find the fdb key to begin the next response from.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>> Hello everyone,
>>>>>>>>>
>>>>>>>>> Based on the discussions on the thread I would like to propose a
>>>>>>> number of first steps:
>>>>>>>>> 1) introduce new endpoints
>>>>>>>>> - {db}/_all_docs/page
>>>>>>>>> - {db}/_all_docs/queries/page
>>>>>>>>> - _all_dbs/page
>>>>>>>>> - _dbs_info/page
>>>>>>>>> - {db}/_design/{ddoc}/_view/{view}/page
>>>>>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
>>>>>>>>> - {db}/_find/page
>>>>>>>>>
>>>>>>>>> These new endpoints would act as follows:
>>>>>>>>> - don't use delayed responses
>>>>>>>>> - return object with following structure
>>>>>>>>> ```
>>>>>>>>> {
>>>>>>>>> "total": Total,
>>>>>>>>> "bookmark": base64 encoded opaque value,
>>>>>>>>> "completed": true | false,
>>>>>>>>> "update_seq": when available,
>>>>>>>>> "page": current page number,
>>>>>>>>> "items": [
>>>>>>>>> ]
>>>>>>>>> }
>>>>>>>>> ```
>>>>>>>>> - the bookmark would include following data (base64 or protobuff???):
>>>>>>>>> - direction
>>>>>>>>> - page
>>>>>>>>> - descending
>>>>>>>>> - endkey
>>>>>>>>> - endkey_docid
>>>>>>>>> - inclusive_end
>>>>>>>>> - startkey
>>>>>>>>> - startkey_docid
>>>>>>>>> - last_key
>>>>>>>>> - update_seq
>>>>>>>>> - timestamp
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> 2) Implement per-endpoint configurable max limits
>>>>>>>>> ```
>>>>>>>>> _all_docs = 5000
>>>>>>>>> _all_docs/queries = 5000
>>>>>>>>> _all_dbs = 5000
>>>>>>>>> _dbs_info = 5000
>>>>>>>>> _view = 2500
>>>>>>>>> _view/queries = 2500
>>>>>>>>> _find = 2500
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> Latter (after few years) CouchDB would deprecate and remove old
>>>>>>> endpoints.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> iilyak
>>>>>>>>>
>>>>>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>>>>>>>>>> Hello everyone,
>>>>>>>>>>
>>>>>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
>>>>>>> CouchDB 4.x
>>>>>>>>>>
>>>>>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>>>>>>>>> read from the database. These are the endpoints I was thinking of:
>>>>>>>>>>
>>>>>>>>>> _all_docs, _all_dbs, _dbs_info and query results
>>>>>>>>>>
>>>>>>>>>> I want to focus on what happens when FoundationDB transactions
>>>>>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>>>>>>>>> feeds, will crash or freeze. The reason is because the
>>>>>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>>>>>>>>> default, so the request handlers run again and end up shoving the
>>>>>>>>>> whole request down the socket again, headers and all, which is
>>>>>>>>>> obviously broken and not what we want.
>>>>>>>>>>
>>>>>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>>>>>>>>> present some behaviors but feel free to add more. Some ideas might
>>>>>>>>>> have been discounted on the IRC discussion already but I'll present
>>>>>>>>>> them anyway in case is sparks further conversation:
>>>>>>>>>>
>>>>>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>>>>>>>>> streaming the data from the next key after last emitted in the
>>>>>>>>>> previous transaction. Document the API behavior change that it may
>>>>>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
>>>>>>>>>> DB.
>>>>>>>>>>
>>>>>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>>>>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>>>>>>>>> - This is the easiest to implement since it would re-use the
>>>>>>>>>> implementation for _changes feed (an extra option passed to the fold
>>>>>>>>>> function).
>>>>>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>>>>>>>>> snapshot view of the data.
>>>>>>>>>>
>>>>>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>>>>>>>>> param which indicates they are aware the stream may end early and so
>>>>>>>>>> would have to paginate from the last emitted key with a skip=1. This
>>>>>>>>>> will keep the request bodies the same as current CouchDB. However, if
>>>>>>>>>> the users got all the data one request, they will end up wasting
>>>>>>>>>> another request to see if there is more data available. If they didn't
>>>>>>>>>> get any data they might have a too large of a skip value (see [2]) so
>>>>>>>>>> would have to guess different values for start/end keys. Or impose max
>>>>>>>>>> limit for the `skip` parameter.
>>>>>>>>>>
>>>>>>>>>> C) End the stream and add a final metadata row like a "transaction":
>>>>>>>>>> "timeout" at the end. That will let the user know to keep paginating
>>>>>>>>>> from the last key onward. This won't work for `_all_dbs` and
>>>>>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>>>>>>>>> feeds and only use this for views and and _all_docs? If we like this
>>>>>>>>>> choice, let's think what happens for those as I couldn't come up with
>>>>>>>>>> anything decent there.
>>>>>>>>>>
>>>>>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>>>>>>>>> "key" of where the iteration stopped and the current "skip" and
>>>>>>>>>> "limit" params, which would keep decreasing. Then user would pass
>>>>>>>>>> those in "start_key=..." in the next request along with the limit and
>>>>>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>>>>>>>>> "key":"..."}. This has the same issue with array results for
>>>>>>>>>> `_all_dbs` and `_dbs_info`[3].
>>>>>>>>>>
>>>>>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>>>>>>>>> there such that response time is likely to fit in one transaction.
>>>>>>>>>> This could be tricky as different runtime environments will have
>>>>>>>>>> different characteristics. Also, if the timeout happens there isn't a
>>>>>>>>>> a nice way to send an HTTP error since we already sent the 200
>>>>>>>>>> response. The downside is that this might break how some users use the
>>>>>>>>>> API, if say the are using large skips and limits already. Perhaps here
>>>>>>>>>> we do both B and D, such that if users want transactional behavior,
>>>>>>>>>> they specify that `transaction=true` param and only then we enforce
>>>>>>>>>> low limit and skip maximums.
>>>>>>>>>>
>>>>>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
>>>>>>>>>> snapshot view doesn't necessarily need to be tied to transaction
>>>>>>>>>> boundaries. We could check the update sequence of the database at the
>>>>>>>>>> start of the next transaction and if it hasn't changed we can continue
>>>>>>>>>> emitting a consistent view. This can apply to C and D and would just
>>>>>>>>>> determine when the stream ends. If there are no writes happening to
>>>>>>>>>> the db, this could potential streams all the data just like option A
>>>>>>>>>> would do. Not entirely sure if this would work for views.
>>>>>>>>>>
>>>>>>>>>> So what do we think? I can see different combinations of options here,
>>>>>>>>>> maybe even different for each API point. For example `_all_dbs`,
>>>>>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>>>>>>>>> have parameters to do F, etc.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> -Nick
>>>>>>>>>>
>>>>>>>>>> Some footnotes:
>>>>>>>>>>
>>>>>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
>>>>>>>>>> per RFC
>>>>>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
>>>>>>> .
>>>>>>>>>> That is, we continue streaming the data by resetting the transaction
>>>>>>>>>> object and restarting from the last emitted key (db sequence in this
>>>>>>>>>> case). However, because the transaction restarts if a document is
>>>>>>>>>> updated while the streaming take place, it may appear in the _changes
>>>>>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>>>>>>>>> have to document it, since previously we presented this point-in-time
>>>>>>>>>> snapshot of the database from when we started streaming.
>>>>>>>>>>
>>>>>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>>>>>>>>> currently support efficient offsets for key selectors
>>>>>>>>>> (
>>>>>>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
>>>>>>> )
>>>>>>>>>> we implemented skip by iterating over the data. This means that a skip
>>>>>>>>>> of say 100000 could keep timing out the transaction without yielding
>>>>>>>>>> any data.
>>>>>>>>>>
>>>>>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>>>>>>>>> obvious place to insert a last metadata row.
>>>>>>>>>>
>>>>>>>>>> [4] For example they have a constraint that documents "a" and "z"
>>>>>>>>>> cannot both be in the database at the same time. But when iterating
>>>>>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
>>>>>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
>>>>>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>>>>>>>>> "relaxed" constrains:
>>>>>>>>>>
>>>>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
> `update_seq` - is same as earlier. Not entirely sure on the intent there.
I just recall another reason why I did this. It helps with etag generation.
I am not going to add it to the spec for now.
On 2020/04/23 21:15:05, Paul Davis <pa...@gmail.com> wrote:
> I'd agree that my initial reaction to cursor was that its not a great
> fit, but there does seem to be the existing name used in the greater
> REST world for this sort of pagination so I'm not concerned about
> using that terminology.
>
> I'm generally on board with allowing and setting some default sane
> limits on pages. We probably should have done that quite awhile ago
> after moving to native clustering and now that we have FDB limits I
> think it makes even more sense to have an API that does not lend
> itself to crazy errors when people are just trying to poke at an API.
>
> I think we're all on board that one of the goals is to make sure that
> clients don't accidentally misinterpret a response. That is, we're
> trying to be quite diligent that a user doesn't get 1000 rows and not
> realize there's another 10 that were beyond the limit. The bookmark
> approach with hard caps seems like a generally fine approach to me.
> The current approach users extra URL path segments to try and avoid
> this confusion. I wonder if we should consider starting to properly
> version our API using one of the many schemes that are used. Having
> read through a few articles I don't have a very clear favorite though.
>
> As to this particular proposal I do see a couple issues:
>
> `total` - We can do this in most cases fairly easily. Though it's a
> bit odd for continuous changes.
>
> `complete` - I'm not sure whether this is entirely possible given the
> API that FDB presents us. Specifically, when we set a range and we get
> back exactly $num_rows in the response, if the data set ended at
> exactly that page I don't think the `more` flag from fdb would tell us
> that. So we'd have a clunky UX there where we say not complete but the
> next page is empty. That's also not to mention that depending on
> whether we're looking at snapshots and so on that there's no way for
> us to know between stateless requests whether there were more rows
> added to the end.
>
> `page` - This one is just hard/impossible to calculate. FDB doesn't
> provide us with offsets or even an efficient "about how many rows in
> this range?" type queries so providing that would be both inaccurate
> and fairly difficult/expensive to calculate. In some cases I think we
> could have something maybe close that didn't suck too badly, but it'd
> also fall down for changes as well due to the way that updates reorder
> them.
>
> `update_seq` - I'm just not sure on when this would be useful or what
> it would refer to. Maybe a version stamp of the last change for that
> request? If we had a future API that asked for a snapshot access then
> maybe? But if we did do something there with versionstamps or read
> versions I'd expect that to come with the rest of the API.
>
> For the bookmark fields:
>
> `direction` vs `descending` seems like a field duplication to me.
>
> `page` - This would seem to suggest we could skip to a certain
> location in the results numerically which we are not able to do with
> the FDB API.
>
> `last_key` vs `start_key` seems like a field duplication. We don't
> need to know where things started I don't think. Just where to start
> from and where to end.
>
> `update_seq` - is same as earlier. Not entirely sure on the intent there.
>
> `timestamp` - Expiring bookmarks based on time does not seem like a
> good idea. Both for clock skew and why bother when this would
> functionally just be a convenience API that users could already
> implement for themselves.
>
> Another thing might also be to provide our bookmark as a full link
> that seems to be fairly standard REST practice these days. Something
> that clients don't have to do any logic with so that we're free to
> change the implementation.
>
> And lastly, I don't think we should be neglecting the _changes API as
> part of this discussion. I realize that we'll need to support the
> older streaming semantics if we want to maintain replication
> compatibility (which I think we'll all agree is a Good Thing) but it
> also feels a bit wrong to ignore it as part of this work if we're
> going to be modernizing our APIs. Though if we do pick up a good
> versioning scheme then we could theoretically make those changes
> easily enough. Plus, who doesn't want to rewrite chttpd to be a whole
> lot less... chttpd-y?
>
>
> On Thu, Apr 23, 2020 at 1:43 PM Robert Samuel Newson <rn...@apache.org> wrote:
> >
> >
> > I think it's a key difference from "cursor" as I've seen them elsewhere, that ours will point at an ever changing database, you couldn't seamlessly cursor through a large data set, one "page" at a time.
> >
> > Bookmarks began in search (raises guilty hand) in order to address a Lucene-specific issue (that high values of "skip" are incredibly inefficient, using lots of RAM). That is not true for CouchDB's own indexes, which can be navigated perfectly with startkey/endkey/startkey_docid/endkey_docid, etc.
> >
> > I guess I'm not helping much with these observations but I wouldn't like to see CouchDB gain an additional and ugly method of doing something already possible.
> >
> > B.
> >
> > > On 23 Apr 2020, at 19:02, Joan Touzet <wo...@apache.org> wrote:
> > >
> > > I realise this is bikeshedding, but I guess that's kind of the point... Everything below is my opinion, not "fact."
> > >
> > > It's unfortunate we need a new endpoint for all of this. In a vacuum I might have just suggested we use the semantics we already have, perhaps with ?from= instead of ?since= .
> > >
> > > "page" only works if the size of a page is well known, either by server preference or directly in the URL. If I ask for:
> > >
> > > GET /{db}/_all_docs?limit=20&page=3
> > >
> > > I know that I'm always going to get document 41 through 60 in the default collation order.
> > >
> > > There's a *fantastic* summary of examples from popular REST APIs here:
> > >
> > > https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
> > >
> > > We are *pretty close* to what a cursor means in those other examples, except for the fact that our cursor can go stale/invalid after a short time.
> > >
> > > Bob, could you be a bit more detailed in your explanation how our definition isn't close to these? Or did you mean SQL CURSOR (which is something entirely different?) If so, I'm fine with this being a REST API cursor - something clearly distinct.
> > >
> > > I come back to wanting to preserve the existing endpoint syntax and naming, without new endpoints, but specifying this new FDB token via ?cursor= and this being the trigger for the new behaviour. At some point, we simply stop accepting ?since= tokens. This seems inline with other popular REST APIs.
> > >
> > > -Joan "still sick and not sleeping right" Touzet
> > >
> > >
> > > On 2020-04-23 12:30, Robert Newson wrote:
> > >> cursor has established meaning in other databases and ours would not be very close to them. I don’t think it’s a good idea.
> > >> B.
> > >>> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
> > >>>
> > >>>
> > >>>>
> > >>>> The best I could come up with is replacing page with
> > >>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> > >>> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
> > >>>
> > >>>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
> > >>>> I agree with Bob that page doesn't make sense as an endpoint. I'm also
> > >>>> rubbish with naming. The best I could come up with is replacing page with
> > >>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> > >>>> All the fields in the bookmark make sense except timestamp. Why would it
> > >>>> matter if the timestamp is old? What happens if a node's time is an hour
> > >>>> behind another node?
> > >>>>
> > >>>>
> > >>>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
> > >>>>>
> > >>>>> - page is to provide some notion of progress for user
> > >>>>> - timestamp - I was thinking that we should drop requests if user would
> > >>>>> try to pass bookmark created an hour ago.
> > >>>>>
> > >>>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
> > >>>>>> "page" and "page number" are odd to me as these don't exist as concepts,
> > >>>>> I'd rather not invent them. I note there's no mention of page size, which
> > >>>>> makes "page number" very vague.
> > >>>>>>
> > >>>>>> What is "timestamp" in the bookmark and what effect does it have when
> > >>>>> the bookmark is passed back in?
> > >>>>>>
> > >>>>>> I guess, why does the bookmark include so much extraneous data? Items
> > >>>>> that are not needed to find the fdb key to begin the next response from.
> > >>>>>>
> > >>>>>>
> > >>>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
> > >>>>>>>
> > >>>>>>> Hello everyone,
> > >>>>>>>
> > >>>>>>> Based on the discussions on the thread I would like to propose a
> > >>>>> number of first steps:
> > >>>>>>> 1) introduce new endpoints
> > >>>>>>> - {db}/_all_docs/page
> > >>>>>>> - {db}/_all_docs/queries/page
> > >>>>>>> - _all_dbs/page
> > >>>>>>> - _dbs_info/page
> > >>>>>>> - {db}/_design/{ddoc}/_view/{view}/page
> > >>>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
> > >>>>>>> - {db}/_find/page
> > >>>>>>>
> > >>>>>>> These new endpoints would act as follows:
> > >>>>>>> - don't use delayed responses
> > >>>>>>> - return object with following structure
> > >>>>>>> ```
> > >>>>>>> {
> > >>>>>>> "total": Total,
> > >>>>>>> "bookmark": base64 encoded opaque value,
> > >>>>>>> "completed": true | false,
> > >>>>>>> "update_seq": when available,
> > >>>>>>> "page": current page number,
> > >>>>>>> "items": [
> > >>>>>>> ]
> > >>>>>>> }
> > >>>>>>> ```
> > >>>>>>> - the bookmark would include following data (base64 or protobuff???):
> > >>>>>>> - direction
> > >>>>>>> - page
> > >>>>>>> - descending
> > >>>>>>> - endkey
> > >>>>>>> - endkey_docid
> > >>>>>>> - inclusive_end
> > >>>>>>> - startkey
> > >>>>>>> - startkey_docid
> > >>>>>>> - last_key
> > >>>>>>> - update_seq
> > >>>>>>> - timestamp
> > >>>>>>> ```
> > >>>>>>>
> > >>>>>>> 2) Implement per-endpoint configurable max limits
> > >>>>>>> ```
> > >>>>>>> _all_docs = 5000
> > >>>>>>> _all_docs/queries = 5000
> > >>>>>>> _all_dbs = 5000
> > >>>>>>> _dbs_info = 5000
> > >>>>>>> _view = 2500
> > >>>>>>> _view/queries = 2500
> > >>>>>>> _find = 2500
> > >>>>>>> ```
> > >>>>>>>
> > >>>>>>> Latter (after few years) CouchDB would deprecate and remove old
> > >>>>> endpoints.
> > >>>>>>>
> > >>>>>>> Best regards,
> > >>>>>>> iilyak
> > >>>>>>>
> > >>>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > >>>>>>>> Hello everyone,
> > >>>>>>>>
> > >>>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
> > >>>>> CouchDB 4.x
> > >>>>>>>>
> > >>>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
> > >>>>>>>> read from the database. These are the endpoints I was thinking of:
> > >>>>>>>>
> > >>>>>>>> _all_docs, _all_dbs, _dbs_info and query results
> > >>>>>>>>
> > >>>>>>>> I want to focus on what happens when FoundationDB transactions
> > >>>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > >>>>>>>> feeds, will crash or freeze. The reason is because the
> > >>>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
> > >>>>>>>> default, so the request handlers run again and end up shoving the
> > >>>>>>>> whole request down the socket again, headers and all, which is
> > >>>>>>>> obviously broken and not what we want.
> > >>>>>>>>
> > >>>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
> > >>>>>>>> present some behaviors but feel free to add more. Some ideas might
> > >>>>>>>> have been discounted on the IRC discussion already but I'll present
> > >>>>>>>> them anyway in case is sparks further conversation:
> > >>>>>>>>
> > >>>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
> > >>>>>>>> streaming the data from the next key after last emitted in the
> > >>>>>>>> previous transaction. Document the API behavior change that it may
> > >>>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
> > >>>>>>>> DB.
> > >>>>>>>>
> > >>>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > >>>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
> > >>>>>>>> - This is the easiest to implement since it would re-use the
> > >>>>>>>> implementation for _changes feed (an extra option passed to the fold
> > >>>>>>>> function).
> > >>>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
> > >>>>>>>> snapshot view of the data.
> > >>>>>>>>
> > >>>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
> > >>>>>>>> param which indicates they are aware the stream may end early and so
> > >>>>>>>> would have to paginate from the last emitted key with a skip=1. This
> > >>>>>>>> will keep the request bodies the same as current CouchDB. However, if
> > >>>>>>>> the users got all the data one request, they will end up wasting
> > >>>>>>>> another request to see if there is more data available. If they didn't
> > >>>>>>>> get any data they might have a too large of a skip value (see [2]) so
> > >>>>>>>> would have to guess different values for start/end keys. Or impose max
> > >>>>>>>> limit for the `skip` parameter.
> > >>>>>>>>
> > >>>>>>>> C) End the stream and add a final metadata row like a "transaction":
> > >>>>>>>> "timeout" at the end. That will let the user know to keep paginating
> > >>>>>>>> from the last key onward. This won't work for `_all_dbs` and
> > >>>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > >>>>>>>> feeds and only use this for views and and _all_docs? If we like this
> > >>>>>>>> choice, let's think what happens for those as I couldn't come up with
> > >>>>>>>> anything decent there.
> > >>>>>>>>
> > >>>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
> > >>>>>>>> "key" of where the iteration stopped and the current "skip" and
> > >>>>>>>> "limit" params, which would keep decreasing. Then user would pass
> > >>>>>>>> those in "start_key=..." in the next request along with the limit and
> > >>>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
> > >>>>>>>> "key":"..."}. This has the same issue with array results for
> > >>>>>>>> `_all_dbs` and `_dbs_info`[3].
> > >>>>>>>>
> > >>>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > >>>>>>>> there such that response time is likely to fit in one transaction.
> > >>>>>>>> This could be tricky as different runtime environments will have
> > >>>>>>>> different characteristics. Also, if the timeout happens there isn't a
> > >>>>>>>> a nice way to send an HTTP error since we already sent the 200
> > >>>>>>>> response. The downside is that this might break how some users use the
> > >>>>>>>> API, if say the are using large skips and limits already. Perhaps here
> > >>>>>>>> we do both B and D, such that if users want transactional behavior,
> > >>>>>>>> they specify that `transaction=true` param and only then we enforce
> > >>>>>>>> low limit and skip maximums.
> > >>>>>>>>
> > >>>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
> > >>>>>>>> snapshot view doesn't necessarily need to be tied to transaction
> > >>>>>>>> boundaries. We could check the update sequence of the database at the
> > >>>>>>>> start of the next transaction and if it hasn't changed we can continue
> > >>>>>>>> emitting a consistent view. This can apply to C and D and would just
> > >>>>>>>> determine when the stream ends. If there are no writes happening to
> > >>>>>>>> the db, this could potential streams all the data just like option A
> > >>>>>>>> would do. Not entirely sure if this would work for views.
> > >>>>>>>>
> > >>>>>>>> So what do we think? I can see different combinations of options here,
> > >>>>>>>> maybe even different for each API point. For example `_all_dbs`,
> > >>>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
> > >>>>>>>> have parameters to do F, etc.
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>> -Nick
> > >>>>>>>>
> > >>>>>>>> Some footnotes:
> > >>>>>>>>
> > >>>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
> > >>>>>>>> per RFC
> > >>>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> > >>>>> .
> > >>>>>>>> That is, we continue streaming the data by resetting the transaction
> > >>>>>>>> object and restarting from the last emitted key (db sequence in this
> > >>>>>>>> case). However, because the transaction restarts if a document is
> > >>>>>>>> updated while the streaming take place, it may appear in the _changes
> > >>>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > >>>>>>>> have to document it, since previously we presented this point-in-time
> > >>>>>>>> snapshot of the database from when we started streaming.
> > >>>>>>>>
> > >>>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > >>>>>>>> currently support efficient offsets for key selectors
> > >>>>>>>> (
> > >>>>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> > >>>>> )
> > >>>>>>>> we implemented skip by iterating over the data. This means that a skip
> > >>>>>>>> of say 100000 could keep timing out the transaction without yielding
> > >>>>>>>> any data.
> > >>>>>>>>
> > >>>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > >>>>>>>> obvious place to insert a last metadata row.
> > >>>>>>>>
> > >>>>>>>> [4] For example they have a constraint that documents "a" and "z"
> > >>>>>>>> cannot both be in the database at the same time. But when iterating
> > >>>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
> > >>>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
> > >>>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
> > >>>>>>>> "relaxed" constrains:
> > >>>>>>>>
> > >>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Paul Davis <pa...@gmail.com>.
I'd agree that my initial reaction to cursor was that its not a great
fit, but there does seem to be the existing name used in the greater
REST world for this sort of pagination so I'm not concerned about
using that terminology.
I'm generally on board with allowing and setting some default sane
limits on pages. We probably should have done that quite awhile ago
after moving to native clustering and now that we have FDB limits I
think it makes even more sense to have an API that does not lend
itself to crazy errors when people are just trying to poke at an API.
I think we're all on board that one of the goals is to make sure that
clients don't accidentally misinterpret a response. That is, we're
trying to be quite diligent that a user doesn't get 1000 rows and not
realize there's another 10 that were beyond the limit. The bookmark
approach with hard caps seems like a generally fine approach to me.
The current approach users extra URL path segments to try and avoid
this confusion. I wonder if we should consider starting to properly
version our API using one of the many schemes that are used. Having
read through a few articles I don't have a very clear favorite though.
As to this particular proposal I do see a couple issues:
`total` - We can do this in most cases fairly easily. Though it's a
bit odd for continuous changes.
`complete` - I'm not sure whether this is entirely possible given the
API that FDB presents us. Specifically, when we set a range and we get
back exactly $num_rows in the response, if the data set ended at
exactly that page I don't think the `more` flag from fdb would tell us
that. So we'd have a clunky UX there where we say not complete but the
next page is empty. That's also not to mention that depending on
whether we're looking at snapshots and so on that there's no way for
us to know between stateless requests whether there were more rows
added to the end.
`page` - This one is just hard/impossible to calculate. FDB doesn't
provide us with offsets or even an efficient "about how many rows in
this range?" type queries so providing that would be both inaccurate
and fairly difficult/expensive to calculate. In some cases I think we
could have something maybe close that didn't suck too badly, but it'd
also fall down for changes as well due to the way that updates reorder
them.
`update_seq` - I'm just not sure on when this would be useful or what
it would refer to. Maybe a version stamp of the last change for that
request? If we had a future API that asked for a snapshot access then
maybe? But if we did do something there with versionstamps or read
versions I'd expect that to come with the rest of the API.
For the bookmark fields:
`direction` vs `descending` seems like a field duplication to me.
`page` - This would seem to suggest we could skip to a certain
location in the results numerically which we are not able to do with
the FDB API.
`last_key` vs `start_key` seems like a field duplication. We don't
need to know where things started I don't think. Just where to start
from and where to end.
`update_seq` - is same as earlier. Not entirely sure on the intent there.
`timestamp` - Expiring bookmarks based on time does not seem like a
good idea. Both for clock skew and why bother when this would
functionally just be a convenience API that users could already
implement for themselves.
Another thing might also be to provide our bookmark as a full link
that seems to be fairly standard REST practice these days. Something
that clients don't have to do any logic with so that we're free to
change the implementation.
And lastly, I don't think we should be neglecting the _changes API as
part of this discussion. I realize that we'll need to support the
older streaming semantics if we want to maintain replication
compatibility (which I think we'll all agree is a Good Thing) but it
also feels a bit wrong to ignore it as part of this work if we're
going to be modernizing our APIs. Though if we do pick up a good
versioning scheme then we could theoretically make those changes
easily enough. Plus, who doesn't want to rewrite chttpd to be a whole
lot less... chttpd-y?
On Thu, Apr 23, 2020 at 1:43 PM Robert Samuel Newson <rn...@apache.org> wrote:
>
>
> I think it's a key difference from "cursor" as I've seen them elsewhere, that ours will point at an ever changing database, you couldn't seamlessly cursor through a large data set, one "page" at a time.
>
> Bookmarks began in search (raises guilty hand) in order to address a Lucene-specific issue (that high values of "skip" are incredibly inefficient, using lots of RAM). That is not true for CouchDB's own indexes, which can be navigated perfectly with startkey/endkey/startkey_docid/endkey_docid, etc.
>
> I guess I'm not helping much with these observations but I wouldn't like to see CouchDB gain an additional and ugly method of doing something already possible.
>
> B.
>
> > On 23 Apr 2020, at 19:02, Joan Touzet <wo...@apache.org> wrote:
> >
> > I realise this is bikeshedding, but I guess that's kind of the point... Everything below is my opinion, not "fact."
> >
> > It's unfortunate we need a new endpoint for all of this. In a vacuum I might have just suggested we use the semantics we already have, perhaps with ?from= instead of ?since= .
> >
> > "page" only works if the size of a page is well known, either by server preference or directly in the URL. If I ask for:
> >
> > GET /{db}/_all_docs?limit=20&page=3
> >
> > I know that I'm always going to get document 41 through 60 in the default collation order.
> >
> > There's a *fantastic* summary of examples from popular REST APIs here:
> >
> > https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
> >
> > We are *pretty close* to what a cursor means in those other examples, except for the fact that our cursor can go stale/invalid after a short time.
> >
> > Bob, could you be a bit more detailed in your explanation how our definition isn't close to these? Or did you mean SQL CURSOR (which is something entirely different?) If so, I'm fine with this being a REST API cursor - something clearly distinct.
> >
> > I come back to wanting to preserve the existing endpoint syntax and naming, without new endpoints, but specifying this new FDB token via ?cursor= and this being the trigger for the new behaviour. At some point, we simply stop accepting ?since= tokens. This seems inline with other popular REST APIs.
> >
> > -Joan "still sick and not sleeping right" Touzet
> >
> >
> > On 2020-04-23 12:30, Robert Newson wrote:
> >> cursor has established meaning in other databases and ours would not be very close to them. I don’t think it’s a good idea.
> >> B.
> >>> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
> >>>
> >>>
> >>>>
> >>>> The best I could come up with is replacing page with
> >>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> >>> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
> >>>
> >>>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
> >>>> I agree with Bob that page doesn't make sense as an endpoint. I'm also
> >>>> rubbish with naming. The best I could come up with is replacing page with
> >>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> >>>> All the fields in the bookmark make sense except timestamp. Why would it
> >>>> matter if the timestamp is old? What happens if a node's time is an hour
> >>>> behind another node?
> >>>>
> >>>>
> >>>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
> >>>>>
> >>>>> - page is to provide some notion of progress for user
> >>>>> - timestamp - I was thinking that we should drop requests if user would
> >>>>> try to pass bookmark created an hour ago.
> >>>>>
> >>>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
> >>>>>> "page" and "page number" are odd to me as these don't exist as concepts,
> >>>>> I'd rather not invent them. I note there's no mention of page size, which
> >>>>> makes "page number" very vague.
> >>>>>>
> >>>>>> What is "timestamp" in the bookmark and what effect does it have when
> >>>>> the bookmark is passed back in?
> >>>>>>
> >>>>>> I guess, why does the bookmark include so much extraneous data? Items
> >>>>> that are not needed to find the fdb key to begin the next response from.
> >>>>>>
> >>>>>>
> >>>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
> >>>>>>>
> >>>>>>> Hello everyone,
> >>>>>>>
> >>>>>>> Based on the discussions on the thread I would like to propose a
> >>>>> number of first steps:
> >>>>>>> 1) introduce new endpoints
> >>>>>>> - {db}/_all_docs/page
> >>>>>>> - {db}/_all_docs/queries/page
> >>>>>>> - _all_dbs/page
> >>>>>>> - _dbs_info/page
> >>>>>>> - {db}/_design/{ddoc}/_view/{view}/page
> >>>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
> >>>>>>> - {db}/_find/page
> >>>>>>>
> >>>>>>> These new endpoints would act as follows:
> >>>>>>> - don't use delayed responses
> >>>>>>> - return object with following structure
> >>>>>>> ```
> >>>>>>> {
> >>>>>>> "total": Total,
> >>>>>>> "bookmark": base64 encoded opaque value,
> >>>>>>> "completed": true | false,
> >>>>>>> "update_seq": when available,
> >>>>>>> "page": current page number,
> >>>>>>> "items": [
> >>>>>>> ]
> >>>>>>> }
> >>>>>>> ```
> >>>>>>> - the bookmark would include following data (base64 or protobuff???):
> >>>>>>> - direction
> >>>>>>> - page
> >>>>>>> - descending
> >>>>>>> - endkey
> >>>>>>> - endkey_docid
> >>>>>>> - inclusive_end
> >>>>>>> - startkey
> >>>>>>> - startkey_docid
> >>>>>>> - last_key
> >>>>>>> - update_seq
> >>>>>>> - timestamp
> >>>>>>> ```
> >>>>>>>
> >>>>>>> 2) Implement per-endpoint configurable max limits
> >>>>>>> ```
> >>>>>>> _all_docs = 5000
> >>>>>>> _all_docs/queries = 5000
> >>>>>>> _all_dbs = 5000
> >>>>>>> _dbs_info = 5000
> >>>>>>> _view = 2500
> >>>>>>> _view/queries = 2500
> >>>>>>> _find = 2500
> >>>>>>> ```
> >>>>>>>
> >>>>>>> Latter (after few years) CouchDB would deprecate and remove old
> >>>>> endpoints.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> iilyak
> >>>>>>>
> >>>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> >>>>>>>> Hello everyone,
> >>>>>>>>
> >>>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
> >>>>> CouchDB 4.x
> >>>>>>>>
> >>>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
> >>>>>>>> read from the database. These are the endpoints I was thinking of:
> >>>>>>>>
> >>>>>>>> _all_docs, _all_dbs, _dbs_info and query results
> >>>>>>>>
> >>>>>>>> I want to focus on what happens when FoundationDB transactions
> >>>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> >>>>>>>> feeds, will crash or freeze. The reason is because the
> >>>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
> >>>>>>>> default, so the request handlers run again and end up shoving the
> >>>>>>>> whole request down the socket again, headers and all, which is
> >>>>>>>> obviously broken and not what we want.
> >>>>>>>>
> >>>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
> >>>>>>>> present some behaviors but feel free to add more. Some ideas might
> >>>>>>>> have been discounted on the IRC discussion already but I'll present
> >>>>>>>> them anyway in case is sparks further conversation:
> >>>>>>>>
> >>>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
> >>>>>>>> streaming the data from the next key after last emitted in the
> >>>>>>>> previous transaction. Document the API behavior change that it may
> >>>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
> >>>>>>>> DB.
> >>>>>>>>
> >>>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> >>>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
> >>>>>>>> - This is the easiest to implement since it would re-use the
> >>>>>>>> implementation for _changes feed (an extra option passed to the fold
> >>>>>>>> function).
> >>>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
> >>>>>>>> snapshot view of the data.
> >>>>>>>>
> >>>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
> >>>>>>>> param which indicates they are aware the stream may end early and so
> >>>>>>>> would have to paginate from the last emitted key with a skip=1. This
> >>>>>>>> will keep the request bodies the same as current CouchDB. However, if
> >>>>>>>> the users got all the data one request, they will end up wasting
> >>>>>>>> another request to see if there is more data available. If they didn't
> >>>>>>>> get any data they might have a too large of a skip value (see [2]) so
> >>>>>>>> would have to guess different values for start/end keys. Or impose max
> >>>>>>>> limit for the `skip` parameter.
> >>>>>>>>
> >>>>>>>> C) End the stream and add a final metadata row like a "transaction":
> >>>>>>>> "timeout" at the end. That will let the user know to keep paginating
> >>>>>>>> from the last key onward. This won't work for `_all_dbs` and
> >>>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> >>>>>>>> feeds and only use this for views and and _all_docs? If we like this
> >>>>>>>> choice, let's think what happens for those as I couldn't come up with
> >>>>>>>> anything decent there.
> >>>>>>>>
> >>>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
> >>>>>>>> "key" of where the iteration stopped and the current "skip" and
> >>>>>>>> "limit" params, which would keep decreasing. Then user would pass
> >>>>>>>> those in "start_key=..." in the next request along with the limit and
> >>>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
> >>>>>>>> "key":"..."}. This has the same issue with array results for
> >>>>>>>> `_all_dbs` and `_dbs_info`[3].
> >>>>>>>>
> >>>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> >>>>>>>> there such that response time is likely to fit in one transaction.
> >>>>>>>> This could be tricky as different runtime environments will have
> >>>>>>>> different characteristics. Also, if the timeout happens there isn't a
> >>>>>>>> a nice way to send an HTTP error since we already sent the 200
> >>>>>>>> response. The downside is that this might break how some users use the
> >>>>>>>> API, if say the are using large skips and limits already. Perhaps here
> >>>>>>>> we do both B and D, such that if users want transactional behavior,
> >>>>>>>> they specify that `transaction=true` param and only then we enforce
> >>>>>>>> low limit and skip maximums.
> >>>>>>>>
> >>>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
> >>>>>>>> snapshot view doesn't necessarily need to be tied to transaction
> >>>>>>>> boundaries. We could check the update sequence of the database at the
> >>>>>>>> start of the next transaction and if it hasn't changed we can continue
> >>>>>>>> emitting a consistent view. This can apply to C and D and would just
> >>>>>>>> determine when the stream ends. If there are no writes happening to
> >>>>>>>> the db, this could potential streams all the data just like option A
> >>>>>>>> would do. Not entirely sure if this would work for views.
> >>>>>>>>
> >>>>>>>> So what do we think? I can see different combinations of options here,
> >>>>>>>> maybe even different for each API point. For example `_all_dbs`,
> >>>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
> >>>>>>>> have parameters to do F, etc.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> -Nick
> >>>>>>>>
> >>>>>>>> Some footnotes:
> >>>>>>>>
> >>>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
> >>>>>>>> per RFC
> >>>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> >>>>> .
> >>>>>>>> That is, we continue streaming the data by resetting the transaction
> >>>>>>>> object and restarting from the last emitted key (db sequence in this
> >>>>>>>> case). However, because the transaction restarts if a document is
> >>>>>>>> updated while the streaming take place, it may appear in the _changes
> >>>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> >>>>>>>> have to document it, since previously we presented this point-in-time
> >>>>>>>> snapshot of the database from when we started streaming.
> >>>>>>>>
> >>>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> >>>>>>>> currently support efficient offsets for key selectors
> >>>>>>>> (
> >>>>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> >>>>> )
> >>>>>>>> we implemented skip by iterating over the data. This means that a skip
> >>>>>>>> of say 100000 could keep timing out the transaction without yielding
> >>>>>>>> any data.
> >>>>>>>>
> >>>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> >>>>>>>> obvious place to insert a last metadata row.
> >>>>>>>>
> >>>>>>>> [4] For example they have a constraint that documents "a" and "z"
> >>>>>>>> cannot both be in the database at the same time. But when iterating
> >>>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
> >>>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
> >>>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
> >>>>>>>> "relaxed" constrains:
> >>>>>>>>
> >>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Robert Samuel Newson <rn...@apache.org>.
I think it's a key difference from "cursor" as I've seen them elsewhere, that ours will point at an ever changing database, you couldn't seamlessly cursor through a large data set, one "page" at a time.
Bookmarks began in search (raises guilty hand) in order to address a Lucene-specific issue (that high values of "skip" are incredibly inefficient, using lots of RAM). That is not true for CouchDB's own indexes, which can be navigated perfectly with startkey/endkey/startkey_docid/endkey_docid, etc.
I guess I'm not helping much with these observations but I wouldn't like to see CouchDB gain an additional and ugly method of doing something already possible.
B.
> On 23 Apr 2020, at 19:02, Joan Touzet <wo...@apache.org> wrote:
>
> I realise this is bikeshedding, but I guess that's kind of the point... Everything below is my opinion, not "fact."
>
> It's unfortunate we need a new endpoint for all of this. In a vacuum I might have just suggested we use the semantics we already have, perhaps with ?from= instead of ?since= .
>
> "page" only works if the size of a page is well known, either by server preference or directly in the URL. If I ask for:
>
> GET /{db}/_all_docs?limit=20&page=3
>
> I know that I'm always going to get document 41 through 60 in the default collation order.
>
> There's a *fantastic* summary of examples from popular REST APIs here:
>
> https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
>
> We are *pretty close* to what a cursor means in those other examples, except for the fact that our cursor can go stale/invalid after a short time.
>
> Bob, could you be a bit more detailed in your explanation how our definition isn't close to these? Or did you mean SQL CURSOR (which is something entirely different?) If so, I'm fine with this being a REST API cursor - something clearly distinct.
>
> I come back to wanting to preserve the existing endpoint syntax and naming, without new endpoints, but specifying this new FDB token via ?cursor= and this being the trigger for the new behaviour. At some point, we simply stop accepting ?since= tokens. This seems inline with other popular REST APIs.
>
> -Joan "still sick and not sleeping right" Touzet
>
>
> On 2020-04-23 12:30, Robert Newson wrote:
>> cursor has established meaning in other databases and ours would not be very close to them. I don’t think it’s a good idea.
>> B.
>>> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
>>>
>>>
>>>>
>>>> The best I could come up with is replacing page with
>>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
>>> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
>>>
>>>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
>>>> I agree with Bob that page doesn't make sense as an endpoint. I'm also
>>>> rubbish with naming. The best I could come up with is replacing page with
>>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
>>>> All the fields in the bookmark make sense except timestamp. Why would it
>>>> matter if the timestamp is old? What happens if a node's time is an hour
>>>> behind another node?
>>>>
>>>>
>>>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
>>>>>
>>>>> - page is to provide some notion of progress for user
>>>>> - timestamp - I was thinking that we should drop requests if user would
>>>>> try to pass bookmark created an hour ago.
>>>>>
>>>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>>> "page" and "page number" are odd to me as these don't exist as concepts,
>>>>> I'd rather not invent them. I note there's no mention of page size, which
>>>>> makes "page number" very vague.
>>>>>>
>>>>>> What is "timestamp" in the bookmark and what effect does it have when
>>>>> the bookmark is passed back in?
>>>>>>
>>>>>> I guess, why does the bookmark include so much extraneous data? Items
>>>>> that are not needed to find the fdb key to begin the next response from.
>>>>>>
>>>>>>
>>>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
>>>>>>>
>>>>>>> Hello everyone,
>>>>>>>
>>>>>>> Based on the discussions on the thread I would like to propose a
>>>>> number of first steps:
>>>>>>> 1) introduce new endpoints
>>>>>>> - {db}/_all_docs/page
>>>>>>> - {db}/_all_docs/queries/page
>>>>>>> - _all_dbs/page
>>>>>>> - _dbs_info/page
>>>>>>> - {db}/_design/{ddoc}/_view/{view}/page
>>>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
>>>>>>> - {db}/_find/page
>>>>>>>
>>>>>>> These new endpoints would act as follows:
>>>>>>> - don't use delayed responses
>>>>>>> - return object with following structure
>>>>>>> ```
>>>>>>> {
>>>>>>> "total": Total,
>>>>>>> "bookmark": base64 encoded opaque value,
>>>>>>> "completed": true | false,
>>>>>>> "update_seq": when available,
>>>>>>> "page": current page number,
>>>>>>> "items": [
>>>>>>> ]
>>>>>>> }
>>>>>>> ```
>>>>>>> - the bookmark would include following data (base64 or protobuff???):
>>>>>>> - direction
>>>>>>> - page
>>>>>>> - descending
>>>>>>> - endkey
>>>>>>> - endkey_docid
>>>>>>> - inclusive_end
>>>>>>> - startkey
>>>>>>> - startkey_docid
>>>>>>> - last_key
>>>>>>> - update_seq
>>>>>>> - timestamp
>>>>>>> ```
>>>>>>>
>>>>>>> 2) Implement per-endpoint configurable max limits
>>>>>>> ```
>>>>>>> _all_docs = 5000
>>>>>>> _all_docs/queries = 5000
>>>>>>> _all_dbs = 5000
>>>>>>> _dbs_info = 5000
>>>>>>> _view = 2500
>>>>>>> _view/queries = 2500
>>>>>>> _find = 2500
>>>>>>> ```
>>>>>>>
>>>>>>> Latter (after few years) CouchDB would deprecate and remove old
>>>>> endpoints.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> iilyak
>>>>>>>
>>>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>>>>>>>> Hello everyone,
>>>>>>>>
>>>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
>>>>> CouchDB 4.x
>>>>>>>>
>>>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>>>>>>> read from the database. These are the endpoints I was thinking of:
>>>>>>>>
>>>>>>>> _all_docs, _all_dbs, _dbs_info and query results
>>>>>>>>
>>>>>>>> I want to focus on what happens when FoundationDB transactions
>>>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>>>>>>> feeds, will crash or freeze. The reason is because the
>>>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>>>>>>> default, so the request handlers run again and end up shoving the
>>>>>>>> whole request down the socket again, headers and all, which is
>>>>>>>> obviously broken and not what we want.
>>>>>>>>
>>>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>>>>>>> present some behaviors but feel free to add more. Some ideas might
>>>>>>>> have been discounted on the IRC discussion already but I'll present
>>>>>>>> them anyway in case is sparks further conversation:
>>>>>>>>
>>>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>>>>>>> streaming the data from the next key after last emitted in the
>>>>>>>> previous transaction. Document the API behavior change that it may
>>>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
>>>>>>>> DB.
>>>>>>>>
>>>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>>>>>>> - This is the easiest to implement since it would re-use the
>>>>>>>> implementation for _changes feed (an extra option passed to the fold
>>>>>>>> function).
>>>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>>>>>>> snapshot view of the data.
>>>>>>>>
>>>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>>>>>>> param which indicates they are aware the stream may end early and so
>>>>>>>> would have to paginate from the last emitted key with a skip=1. This
>>>>>>>> will keep the request bodies the same as current CouchDB. However, if
>>>>>>>> the users got all the data one request, they will end up wasting
>>>>>>>> another request to see if there is more data available. If they didn't
>>>>>>>> get any data they might have a too large of a skip value (see [2]) so
>>>>>>>> would have to guess different values for start/end keys. Or impose max
>>>>>>>> limit for the `skip` parameter.
>>>>>>>>
>>>>>>>> C) End the stream and add a final metadata row like a "transaction":
>>>>>>>> "timeout" at the end. That will let the user know to keep paginating
>>>>>>>> from the last key onward. This won't work for `_all_dbs` and
>>>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>>>>>>> feeds and only use this for views and and _all_docs? If we like this
>>>>>>>> choice, let's think what happens for those as I couldn't come up with
>>>>>>>> anything decent there.
>>>>>>>>
>>>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>>>>>>> "key" of where the iteration stopped and the current "skip" and
>>>>>>>> "limit" params, which would keep decreasing. Then user would pass
>>>>>>>> those in "start_key=..." in the next request along with the limit and
>>>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>>>>>>> "key":"..."}. This has the same issue with array results for
>>>>>>>> `_all_dbs` and `_dbs_info`[3].
>>>>>>>>
>>>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>>>>>>> there such that response time is likely to fit in one transaction.
>>>>>>>> This could be tricky as different runtime environments will have
>>>>>>>> different characteristics. Also, if the timeout happens there isn't a
>>>>>>>> a nice way to send an HTTP error since we already sent the 200
>>>>>>>> response. The downside is that this might break how some users use the
>>>>>>>> API, if say the are using large skips and limits already. Perhaps here
>>>>>>>> we do both B and D, such that if users want transactional behavior,
>>>>>>>> they specify that `transaction=true` param and only then we enforce
>>>>>>>> low limit and skip maximums.
>>>>>>>>
>>>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
>>>>>>>> snapshot view doesn't necessarily need to be tied to transaction
>>>>>>>> boundaries. We could check the update sequence of the database at the
>>>>>>>> start of the next transaction and if it hasn't changed we can continue
>>>>>>>> emitting a consistent view. This can apply to C and D and would just
>>>>>>>> determine when the stream ends. If there are no writes happening to
>>>>>>>> the db, this could potential streams all the data just like option A
>>>>>>>> would do. Not entirely sure if this would work for views.
>>>>>>>>
>>>>>>>> So what do we think? I can see different combinations of options here,
>>>>>>>> maybe even different for each API point. For example `_all_dbs`,
>>>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>>>>>>> have parameters to do F, etc.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> -Nick
>>>>>>>>
>>>>>>>> Some footnotes:
>>>>>>>>
>>>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
>>>>>>>> per RFC
>>>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
>>>>> .
>>>>>>>> That is, we continue streaming the data by resetting the transaction
>>>>>>>> object and restarting from the last emitted key (db sequence in this
>>>>>>>> case). However, because the transaction restarts if a document is
>>>>>>>> updated while the streaming take place, it may appear in the _changes
>>>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>>>>>>> have to document it, since previously we presented this point-in-time
>>>>>>>> snapshot of the database from when we started streaming.
>>>>>>>>
>>>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>>>>>>> currently support efficient offsets for key selectors
>>>>>>>> (
>>>>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
>>>>> )
>>>>>>>> we implemented skip by iterating over the data. This means that a skip
>>>>>>>> of say 100000 could keep timing out the transaction without yielding
>>>>>>>> any data.
>>>>>>>>
>>>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>>>>>>> obvious place to insert a last metadata row.
>>>>>>>>
>>>>>>>> [4] For example they have a constraint that documents "a" and "z"
>>>>>>>> cannot both be in the database at the same time. But when iterating
>>>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
>>>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
>>>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>>>>>>> "relaxed" constrains:
>>>>>>>>
>>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Joan Touzet <wo...@apache.org>.
I realise this is bikeshedding, but I guess that's kind of the point...
Everything below is my opinion, not "fact."
It's unfortunate we need a new endpoint for all of this. In a vacuum I
might have just suggested we use the semantics we already have, perhaps
with ?from= instead of ?since= .
"page" only works if the size of a page is well known, either by server
preference or directly in the URL. If I ask for:
GET /{db}/_all_docs?limit=20&page=3
I know that I'm always going to get document 41 through 60 in the
default collation order.
There's a *fantastic* summary of examples from popular REST APIs here:
https://medium.com/@ignaciochiazzo/paginating-requests-in-apis-d4883d4c1c4c
We are *pretty close* to what a cursor means in those other examples,
except for the fact that our cursor can go stale/invalid after a short time.
Bob, could you be a bit more detailed in your explanation how our
definition isn't close to these? Or did you mean SQL CURSOR (which is
something entirely different?) If so, I'm fine with this being a REST
API cursor - something clearly distinct.
I come back to wanting to preserve the existing endpoint syntax and
naming, without new endpoints, but specifying this new FDB token via
?cursor= and this being the trigger for the new behaviour. At some
point, we simply stop accepting ?since= tokens. This seems inline with
other popular REST APIs.
-Joan "still sick and not sleeping right" Touzet
On 2020-04-23 12:30, Robert Newson wrote:
> cursor has established meaning in other databases and ours would not be very close to them. I don’t think it’s a good idea.
>
> B.
>
>> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
>>
>>
>>>
>>> The best I could come up with is replacing page with
>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
>> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
>>
>>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
>>> I agree with Bob that page doesn't make sense as an endpoint. I'm also
>>> rubbish with naming. The best I could come up with is replacing page with
>>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
>>> All the fields in the bookmark make sense except timestamp. Why would it
>>> matter if the timestamp is old? What happens if a node's time is an hour
>>> behind another node?
>>>
>>>
>>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
>>>>
>>>> - page is to provide some notion of progress for user
>>>> - timestamp - I was thinking that we should drop requests if user would
>>>> try to pass bookmark created an hour ago.
>>>>
>>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>> "page" and "page number" are odd to me as these don't exist as concepts,
>>>> I'd rather not invent them. I note there's no mention of page size, which
>>>> makes "page number" very vague.
>>>>>
>>>>> What is "timestamp" in the bookmark and what effect does it have when
>>>> the bookmark is passed back in?
>>>>>
>>>>> I guess, why does the bookmark include so much extraneous data? Items
>>>> that are not needed to find the fdb key to begin the next response from.
>>>>>
>>>>>
>>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
>>>>>>
>>>>>> Hello everyone,
>>>>>>
>>>>>> Based on the discussions on the thread I would like to propose a
>>>> number of first steps:
>>>>>> 1) introduce new endpoints
>>>>>> - {db}/_all_docs/page
>>>>>> - {db}/_all_docs/queries/page
>>>>>> - _all_dbs/page
>>>>>> - _dbs_info/page
>>>>>> - {db}/_design/{ddoc}/_view/{view}/page
>>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
>>>>>> - {db}/_find/page
>>>>>>
>>>>>> These new endpoints would act as follows:
>>>>>> - don't use delayed responses
>>>>>> - return object with following structure
>>>>>> ```
>>>>>> {
>>>>>> "total": Total,
>>>>>> "bookmark": base64 encoded opaque value,
>>>>>> "completed": true | false,
>>>>>> "update_seq": when available,
>>>>>> "page": current page number,
>>>>>> "items": [
>>>>>> ]
>>>>>> }
>>>>>> ```
>>>>>> - the bookmark would include following data (base64 or protobuff???):
>>>>>> - direction
>>>>>> - page
>>>>>> - descending
>>>>>> - endkey
>>>>>> - endkey_docid
>>>>>> - inclusive_end
>>>>>> - startkey
>>>>>> - startkey_docid
>>>>>> - last_key
>>>>>> - update_seq
>>>>>> - timestamp
>>>>>> ```
>>>>>>
>>>>>> 2) Implement per-endpoint configurable max limits
>>>>>> ```
>>>>>> _all_docs = 5000
>>>>>> _all_docs/queries = 5000
>>>>>> _all_dbs = 5000
>>>>>> _dbs_info = 5000
>>>>>> _view = 2500
>>>>>> _view/queries = 2500
>>>>>> _find = 2500
>>>>>> ```
>>>>>>
>>>>>> Latter (after few years) CouchDB would deprecate and remove old
>>>> endpoints.
>>>>>>
>>>>>> Best regards,
>>>>>> iilyak
>>>>>>
>>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>>>>>>> Hello everyone,
>>>>>>>
>>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
>>>> CouchDB 4.x
>>>>>>>
>>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>>>>>> read from the database. These are the endpoints I was thinking of:
>>>>>>>
>>>>>>> _all_docs, _all_dbs, _dbs_info and query results
>>>>>>>
>>>>>>> I want to focus on what happens when FoundationDB transactions
>>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>>>>>> feeds, will crash or freeze. The reason is because the
>>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>>>>>> default, so the request handlers run again and end up shoving the
>>>>>>> whole request down the socket again, headers and all, which is
>>>>>>> obviously broken and not what we want.
>>>>>>>
>>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>>>>>> present some behaviors but feel free to add more. Some ideas might
>>>>>>> have been discounted on the IRC discussion already but I'll present
>>>>>>> them anyway in case is sparks further conversation:
>>>>>>>
>>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>>>>>> streaming the data from the next key after last emitted in the
>>>>>>> previous transaction. Document the API behavior change that it may
>>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
>>>>>>> DB.
>>>>>>>
>>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>>>>>> - This is the easiest to implement since it would re-use the
>>>>>>> implementation for _changes feed (an extra option passed to the fold
>>>>>>> function).
>>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>>>>>> snapshot view of the data.
>>>>>>>
>>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>>>>>> param which indicates they are aware the stream may end early and so
>>>>>>> would have to paginate from the last emitted key with a skip=1. This
>>>>>>> will keep the request bodies the same as current CouchDB. However, if
>>>>>>> the users got all the data one request, they will end up wasting
>>>>>>> another request to see if there is more data available. If they didn't
>>>>>>> get any data they might have a too large of a skip value (see [2]) so
>>>>>>> would have to guess different values for start/end keys. Or impose max
>>>>>>> limit for the `skip` parameter.
>>>>>>>
>>>>>>> C) End the stream and add a final metadata row like a "transaction":
>>>>>>> "timeout" at the end. That will let the user know to keep paginating
>>>>>>> from the last key onward. This won't work for `_all_dbs` and
>>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>>>>>> feeds and only use this for views and and _all_docs? If we like this
>>>>>>> choice, let's think what happens for those as I couldn't come up with
>>>>>>> anything decent there.
>>>>>>>
>>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>>>>>> "key" of where the iteration stopped and the current "skip" and
>>>>>>> "limit" params, which would keep decreasing. Then user would pass
>>>>>>> those in "start_key=..." in the next request along with the limit and
>>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>>>>>> "key":"..."}. This has the same issue with array results for
>>>>>>> `_all_dbs` and `_dbs_info`[3].
>>>>>>>
>>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>>>>>> there such that response time is likely to fit in one transaction.
>>>>>>> This could be tricky as different runtime environments will have
>>>>>>> different characteristics. Also, if the timeout happens there isn't a
>>>>>>> a nice way to send an HTTP error since we already sent the 200
>>>>>>> response. The downside is that this might break how some users use the
>>>>>>> API, if say the are using large skips and limits already. Perhaps here
>>>>>>> we do both B and D, such that if users want transactional behavior,
>>>>>>> they specify that `transaction=true` param and only then we enforce
>>>>>>> low limit and skip maximums.
>>>>>>>
>>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
>>>>>>> snapshot view doesn't necessarily need to be tied to transaction
>>>>>>> boundaries. We could check the update sequence of the database at the
>>>>>>> start of the next transaction and if it hasn't changed we can continue
>>>>>>> emitting a consistent view. This can apply to C and D and would just
>>>>>>> determine when the stream ends. If there are no writes happening to
>>>>>>> the db, this could potential streams all the data just like option A
>>>>>>> would do. Not entirely sure if this would work for views.
>>>>>>>
>>>>>>> So what do we think? I can see different combinations of options here,
>>>>>>> maybe even different for each API point. For example `_all_dbs`,
>>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>>>>>> have parameters to do F, etc.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> -Nick
>>>>>>>
>>>>>>> Some footnotes:
>>>>>>>
>>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
>>>>>>> per RFC
>>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
>>>> .
>>>>>>> That is, we continue streaming the data by resetting the transaction
>>>>>>> object and restarting from the last emitted key (db sequence in this
>>>>>>> case). However, because the transaction restarts if a document is
>>>>>>> updated while the streaming take place, it may appear in the _changes
>>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>>>>>> have to document it, since previously we presented this point-in-time
>>>>>>> snapshot of the database from when we started streaming.
>>>>>>>
>>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>>>>>> currently support efficient offsets for key selectors
>>>>>>> (
>>>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
>>>> )
>>>>>>> we implemented skip by iterating over the data. This means that a skip
>>>>>>> of say 100000 could keep timing out the transaction without yielding
>>>>>>> any data.
>>>>>>>
>>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>>>>>> obvious place to insert a last metadata row.
>>>>>>>
>>>>>>> [4] For example they have a constraint that documents "a" and "z"
>>>>>>> cannot both be in the database at the same time. But when iterating
>>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
>>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
>>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>>>>>> "relaxed" constrains:
>>>>>>>
>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Robert Newson <bo...@rsn.io>.
cursor has established meaning in other databases and ours would not be very close to them. I don’t think it’s a good idea.
B.
> On 23 Apr 2020, at 11:50, Ilya Khlopotov <ii...@apache.org> wrote:
>
>
>>
>> The best I could come up with is replacing page with
>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
>
>> On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
>> I agree with Bob that page doesn't make sense as an endpoint. I'm also
>> rubbish with naming. The best I could come up with is replacing page with
>> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
>> All the fields in the bookmark make sense except timestamp. Why would it
>> matter if the timestamp is old? What happens if a node's time is an hour
>> behind another node?
>>
>>
>>> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
>>>
>>> - page is to provide some notion of progress for user
>>> - timestamp - I was thinking that we should drop requests if user would
>>> try to pass bookmark created an hour ago.
>>>
>>> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
>>>> "page" and "page number" are odd to me as these don't exist as concepts,
>>> I'd rather not invent them. I note there's no mention of page size, which
>>> makes "page number" very vague.
>>>>
>>>> What is "timestamp" in the bookmark and what effect does it have when
>>> the bookmark is passed back in?
>>>>
>>>> I guess, why does the bookmark include so much extraneous data? Items
>>> that are not needed to find the fdb key to begin the next response from.
>>>>
>>>>
>>>>> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
>>>>>
>>>>> Hello everyone,
>>>>>
>>>>> Based on the discussions on the thread I would like to propose a
>>> number of first steps:
>>>>> 1) introduce new endpoints
>>>>> - {db}/_all_docs/page
>>>>> - {db}/_all_docs/queries/page
>>>>> - _all_dbs/page
>>>>> - _dbs_info/page
>>>>> - {db}/_design/{ddoc}/_view/{view}/page
>>>>> - {db}/_design/{ddoc}/_view/{view}/queries/page
>>>>> - {db}/_find/page
>>>>>
>>>>> These new endpoints would act as follows:
>>>>> - don't use delayed responses
>>>>> - return object with following structure
>>>>> ```
>>>>> {
>>>>> "total": Total,
>>>>> "bookmark": base64 encoded opaque value,
>>>>> "completed": true | false,
>>>>> "update_seq": when available,
>>>>> "page": current page number,
>>>>> "items": [
>>>>> ]
>>>>> }
>>>>> ```
>>>>> - the bookmark would include following data (base64 or protobuff???):
>>>>> - direction
>>>>> - page
>>>>> - descending
>>>>> - endkey
>>>>> - endkey_docid
>>>>> - inclusive_end
>>>>> - startkey
>>>>> - startkey_docid
>>>>> - last_key
>>>>> - update_seq
>>>>> - timestamp
>>>>> ```
>>>>>
>>>>> 2) Implement per-endpoint configurable max limits
>>>>> ```
>>>>> _all_docs = 5000
>>>>> _all_docs/queries = 5000
>>>>> _all_dbs = 5000
>>>>> _dbs_info = 5000
>>>>> _view = 2500
>>>>> _view/queries = 2500
>>>>> _find = 2500
>>>>> ```
>>>>>
>>>>> Latter (after few years) CouchDB would deprecate and remove old
>>> endpoints.
>>>>>
>>>>> Best regards,
>>>>> iilyak
>>>>>
>>>>> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>>>>>> Hello everyone,
>>>>>>
>>>>>> I'd like to discuss the shape and behavior of streaming APIs for
>>> CouchDB 4.x
>>>>>>
>>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>>>>> read from the database. These are the endpoints I was thinking of:
>>>>>>
>>>>>> _all_docs, _all_dbs, _dbs_info and query results
>>>>>>
>>>>>> I want to focus on what happens when FoundationDB transactions
>>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>>>>> feeds, will crash or freeze. The reason is because the
>>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>>>>> default, so the request handlers run again and end up shoving the
>>>>>> whole request down the socket again, headers and all, which is
>>>>>> obviously broken and not what we want.
>>>>>>
>>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>>>>> present some behaviors but feel free to add more. Some ideas might
>>>>>> have been discounted on the IRC discussion already but I'll present
>>>>>> them anyway in case is sparks further conversation:
>>>>>>
>>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>>>>> streaming the data from the next key after last emitted in the
>>>>>> previous transaction. Document the API behavior change that it may
>>>>>> present a view of the data is never a point-in-time[4] snapshot of the
>>>>>> DB.
>>>>>>
>>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>>>>> - This is the easiest to implement since it would re-use the
>>>>>> implementation for _changes feed (an extra option passed to the fold
>>>>>> function).
>>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>>>>> snapshot view of the data.
>>>>>>
>>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>>>>> param which indicates they are aware the stream may end early and so
>>>>>> would have to paginate from the last emitted key with a skip=1. This
>>>>>> will keep the request bodies the same as current CouchDB. However, if
>>>>>> the users got all the data one request, they will end up wasting
>>>>>> another request to see if there is more data available. If they didn't
>>>>>> get any data they might have a too large of a skip value (see [2]) so
>>>>>> would have to guess different values for start/end keys. Or impose max
>>>>>> limit for the `skip` parameter.
>>>>>>
>>>>>> C) End the stream and add a final metadata row like a "transaction":
>>>>>> "timeout" at the end. That will let the user know to keep paginating
>>>>>> from the last key onward. This won't work for `_all_dbs` and
>>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>>>>> feeds and only use this for views and and _all_docs? If we like this
>>>>>> choice, let's think what happens for those as I couldn't come up with
>>>>>> anything decent there.
>>>>>>
>>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>>>>> "key" of where the iteration stopped and the current "skip" and
>>>>>> "limit" params, which would keep decreasing. Then user would pass
>>>>>> those in "start_key=..." in the next request along with the limit and
>>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>>>>> "key":"..."}. This has the same issue with array results for
>>>>>> `_all_dbs` and `_dbs_info`[3].
>>>>>>
>>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>>>>> there such that response time is likely to fit in one transaction.
>>>>>> This could be tricky as different runtime environments will have
>>>>>> different characteristics. Also, if the timeout happens there isn't a
>>>>>> a nice way to send an HTTP error since we already sent the 200
>>>>>> response. The downside is that this might break how some users use the
>>>>>> API, if say the are using large skips and limits already. Perhaps here
>>>>>> we do both B and D, such that if users want transactional behavior,
>>>>>> they specify that `transaction=true` param and only then we enforce
>>>>>> low limit and skip maximums.
>>>>>>
>>>>>> F) At least for `_all_docs` it seems providing a point-in-time
>>>>>> snapshot view doesn't necessarily need to be tied to transaction
>>>>>> boundaries. We could check the update sequence of the database at the
>>>>>> start of the next transaction and if it hasn't changed we can continue
>>>>>> emitting a consistent view. This can apply to C and D and would just
>>>>>> determine when the stream ends. If there are no writes happening to
>>>>>> the db, this could potential streams all the data just like option A
>>>>>> would do. Not entirely sure if this would work for views.
>>>>>>
>>>>>> So what do we think? I can see different combinations of options here,
>>>>>> maybe even different for each API point. For example `_all_dbs`,
>>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>>>>> have parameters to do F, etc.
>>>>>>
>>>>>> Cheers,
>>>>>> -Nick
>>>>>>
>>>>>> Some footnotes:
>>>>>>
>>>>>> [1] _changes feeds is the only one that works currently. It behaves as
>>>>>> per RFC
>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
>>> .
>>>>>> That is, we continue streaming the data by resetting the transaction
>>>>>> object and restarting from the last emitted key (db sequence in this
>>>>>> case). However, because the transaction restarts if a document is
>>>>>> updated while the streaming take place, it may appear in the _changes
>>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>>>>> have to document it, since previously we presented this point-in-time
>>>>>> snapshot of the database from when we started streaming.
>>>>>>
>>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>>>>> currently support efficient offsets for key selectors
>>>>>> (
>>> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
>>> )
>>>>>> we implemented skip by iterating over the data. This means that a skip
>>>>>> of say 100000 could keep timing out the transaction without yielding
>>>>>> any data.
>>>>>>
>>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>>>>> obvious place to insert a last metadata row.
>>>>>>
>>>>>> [4] For example they have a constraint that documents "a" and "z"
>>>>>> cannot both be in the database at the same time. But when iterating
>>>>>> it's possible that "a" was there at the start. Then by the end, "a"
>>>>>> was removed and "z" added, so both "a" and "z" would appear in the
>>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>>>>> "relaxed" constrains:
>>>>>>
>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>>>>
>>>>
>>>>
>>>
>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
> The best I could come up with is replacing page with
> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
Good idea, I like {db}/_all_docs/cursor (or {db}/_all_docs/_cursor).
On 2020/04/23 08:54:36, Garren Smith <ga...@apache.org> wrote:
> I agree with Bob that page doesn't make sense as an endpoint. I'm also
> rubbish with naming. The best I could come up with is replacing page with
> cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
> All the fields in the bookmark make sense except timestamp. Why would it
> matter if the timestamp is old? What happens if a node's time is an hour
> behind another node?
>
>
> On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
>
> > - page is to provide some notion of progress for user
> > - timestamp - I was thinking that we should drop requests if user would
> > try to pass bookmark created an hour ago.
> >
> > On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
> > > "page" and "page number" are odd to me as these don't exist as concepts,
> > I'd rather not invent them. I note there's no mention of page size, which
> > makes "page number" very vague.
> > >
> > > What is "timestamp" in the bookmark and what effect does it have when
> > the bookmark is passed back in?
> > >
> > > I guess, why does the bookmark include so much extraneous data? Items
> > that are not needed to find the fdb key to begin the next response from.
> > >
> > >
> > > > On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > Based on the discussions on the thread I would like to propose a
> > number of first steps:
> > > > 1) introduce new endpoints
> > > > - {db}/_all_docs/page
> > > > - {db}/_all_docs/queries/page
> > > > - _all_dbs/page
> > > > - _dbs_info/page
> > > > - {db}/_design/{ddoc}/_view/{view}/page
> > > > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > > - {db}/_find/page
> > > >
> > > > These new endpoints would act as follows:
> > > > - don't use delayed responses
> > > > - return object with following structure
> > > > ```
> > > > {
> > > > "total": Total,
> > > > "bookmark": base64 encoded opaque value,
> > > > "completed": true | false,
> > > > "update_seq": when available,
> > > > "page": current page number,
> > > > "items": [
> > > > ]
> > > > }
> > > > ```
> > > > - the bookmark would include following data (base64 or protobuff???):
> > > > - direction
> > > > - page
> > > > - descending
> > > > - endkey
> > > > - endkey_docid
> > > > - inclusive_end
> > > > - startkey
> > > > - startkey_docid
> > > > - last_key
> > > > - update_seq
> > > > - timestamp
> > > > ```
> > > >
> > > > 2) Implement per-endpoint configurable max limits
> > > > ```
> > > > _all_docs = 5000
> > > > _all_docs/queries = 5000
> > > > _all_dbs = 5000
> > > > _dbs_info = 5000
> > > > _view = 2500
> > > > _view/queries = 2500
> > > > _find = 2500
> > > > ```
> > > >
> > > > Latter (after few years) CouchDB would deprecate and remove old
> > endpoints.
> > > >
> > > > Best regards,
> > > > iilyak
> > > >
> > > > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > > >> Hello everyone,
> > > >>
> > > >> I'd like to discuss the shape and behavior of streaming APIs for
> > CouchDB 4.x
> > > >>
> > > >> By "streaming APIs" I mean APIs which stream data in row as it gets
> > > >> read from the database. These are the endpoints I was thinking of:
> > > >>
> > > >> _all_docs, _all_dbs, _dbs_info and query results
> > > >>
> > > >> I want to focus on what happens when FoundationDB transactions
> > > >> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > > >> feeds, will crash or freeze. The reason is because the
> > > >> transaction_too_old error at the end of 5 seconds is retry-able by
> > > >> default, so the request handlers run again and end up shoving the
> > > >> whole request down the socket again, headers and all, which is
> > > >> obviously broken and not what we want.
> > > >>
> > > >> There are few alternatives discussed in couchdb-dev channel. I'll
> > > >> present some behaviors but feel free to add more. Some ideas might
> > > >> have been discounted on the IRC discussion already but I'll present
> > > >> them anyway in case is sparks further conversation:
> > > >>
> > > >> A) Do what _changes[1] feeds do. Start a new transaction and continue
> > > >> streaming the data from the next key after last emitted in the
> > > >> previous transaction. Document the API behavior change that it may
> > > >> present a view of the data is never a point-in-time[4] snapshot of the
> > > >> DB.
> > > >>
> > > >> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > > >> don't have to change to continue using these CouchDB 4.0 endpoints
> > > >> - This is the easiest to implement since it would re-use the
> > > >> implementation for _changes feed (an extra option passed to the fold
> > > >> function).
> > > >> - Breaks API behavior if users relied on having a point-in-time[4]
> > > >> snapshot view of the data.
> > > >>
> > > >> B) Simply end the stream. Let the users pass a `?transaction=true`
> > > >> param which indicates they are aware the stream may end early and so
> > > >> would have to paginate from the last emitted key with a skip=1. This
> > > >> will keep the request bodies the same as current CouchDB. However, if
> > > >> the users got all the data one request, they will end up wasting
> > > >> another request to see if there is more data available. If they didn't
> > > >> get any data they might have a too large of a skip value (see [2]) so
> > > >> would have to guess different values for start/end keys. Or impose max
> > > >> limit for the `skip` parameter.
> > > >>
> > > >> C) End the stream and add a final metadata row like a "transaction":
> > > >> "timeout" at the end. That will let the user know to keep paginating
> > > >> from the last key onward. This won't work for `_all_dbs` and
> > > >> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > > >> feeds and only use this for views and and _all_docs? If we like this
> > > >> choice, let's think what happens for those as I couldn't come up with
> > > >> anything decent there.
> > > >>
> > > >> D) Same as C but to solve the issue with skips[2], emit a bookmark
> > > >> "key" of where the iteration stopped and the current "skip" and
> > > >> "limit" params, which would keep decreasing. Then user would pass
> > > >> those in "start_key=..." in the next request along with the limit and
> > > >> skip params. So something like "continuation":{"skip":599, "limit":5,
> > > >> "key":"..."}. This has the same issue with array results for
> > > >> `_all_dbs` and `_dbs_info`[3].
> > > >>
> > > >> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > > >> there such that response time is likely to fit in one transaction.
> > > >> This could be tricky as different runtime environments will have
> > > >> different characteristics. Also, if the timeout happens there isn't a
> > > >> a nice way to send an HTTP error since we already sent the 200
> > > >> response. The downside is that this might break how some users use the
> > > >> API, if say the are using large skips and limits already. Perhaps here
> > > >> we do both B and D, such that if users want transactional behavior,
> > > >> they specify that `transaction=true` param and only then we enforce
> > > >> low limit and skip maximums.
> > > >>
> > > >> F) At least for `_all_docs` it seems providing a point-in-time
> > > >> snapshot view doesn't necessarily need to be tied to transaction
> > > >> boundaries. We could check the update sequence of the database at the
> > > >> start of the next transaction and if it hasn't changed we can continue
> > > >> emitting a consistent view. This can apply to C and D and would just
> > > >> determine when the stream ends. If there are no writes happening to
> > > >> the db, this could potential streams all the data just like option A
> > > >> would do. Not entirely sure if this would work for views.
> > > >>
> > > >> So what do we think? I can see different combinations of options here,
> > > >> maybe even different for each API point. For example `_all_dbs`,
> > > >> `_dbs_info` are always A, and `_all_docs` and views default to A but
> > > >> have parameters to do F, etc.
> > > >>
> > > >> Cheers,
> > > >> -Nick
> > > >>
> > > >> Some footnotes:
> > > >>
> > > >> [1] _changes feeds is the only one that works currently. It behaves as
> > > >> per RFC
> > https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> > .
> > > >> That is, we continue streaming the data by resetting the transaction
> > > >> object and restarting from the last emitted key (db sequence in this
> > > >> case). However, because the transaction restarts if a document is
> > > >> updated while the streaming take place, it may appear in the _changes
> > > >> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > > >> have to document it, since previously we presented this point-in-time
> > > >> snapshot of the database from when we started streaming.
> > > >>
> > > >> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > > >> currently support efficient offsets for key selectors
> > > >> (
> > https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> > )
> > > >> we implemented skip by iterating over the data. This means that a skip
> > > >> of say 100000 could keep timing out the transaction without yielding
> > > >> any data.
> > > >>
> > > >> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > > >> obvious place to insert a last metadata row.
> > > >>
> > > >> [4] For example they have a constraint that documents "a" and "z"
> > > >> cannot both be in the database at the same time. But when iterating
> > > >> it's possible that "a" was there at the start. Then by the end, "a"
> > > >> was removed and "z" added, so both "a" and "z" would appear in the
> > > >> emitted stream. Note that FoundationDB has APIs which exhibit the same
> > > >> "relaxed" constrains:
> > > >>
> > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Garren Smith <ga...@apache.org>.
I agree with Bob that page doesn't make sense as an endpoint. I'm also
rubbish with naming. The best I could come up with is replacing page with
cursor - {db}/_all_docs/cursor or possibly {db}/_cursor/_all_docs
All the fields in the bookmark make sense except timestamp. Why would it
matter if the timestamp is old? What happens if a node's time is an hour
behind another node?
On Thu, Apr 23, 2020 at 4:55 AM Ilya Khlopotov <ii...@apache.org> wrote:
> - page is to provide some notion of progress for user
> - timestamp - I was thinking that we should drop requests if user would
> try to pass bookmark created an hour ago.
>
> On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
> > "page" and "page number" are odd to me as these don't exist as concepts,
> I'd rather not invent them. I note there's no mention of page size, which
> makes "page number" very vague.
> >
> > What is "timestamp" in the bookmark and what effect does it have when
> the bookmark is passed back in?
> >
> > I guess, why does the bookmark include so much extraneous data? Items
> that are not needed to find the fdb key to begin the next response from.
> >
> >
> > > On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
> > >
> > > Hello everyone,
> > >
> > > Based on the discussions on the thread I would like to propose a
> number of first steps:
> > > 1) introduce new endpoints
> > > - {db}/_all_docs/page
> > > - {db}/_all_docs/queries/page
> > > - _all_dbs/page
> > > - _dbs_info/page
> > > - {db}/_design/{ddoc}/_view/{view}/page
> > > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > > - {db}/_find/page
> > >
> > > These new endpoints would act as follows:
> > > - don't use delayed responses
> > > - return object with following structure
> > > ```
> > > {
> > > "total": Total,
> > > "bookmark": base64 encoded opaque value,
> > > "completed": true | false,
> > > "update_seq": when available,
> > > "page": current page number,
> > > "items": [
> > > ]
> > > }
> > > ```
> > > - the bookmark would include following data (base64 or protobuff???):
> > > - direction
> > > - page
> > > - descending
> > > - endkey
> > > - endkey_docid
> > > - inclusive_end
> > > - startkey
> > > - startkey_docid
> > > - last_key
> > > - update_seq
> > > - timestamp
> > > ```
> > >
> > > 2) Implement per-endpoint configurable max limits
> > > ```
> > > _all_docs = 5000
> > > _all_docs/queries = 5000
> > > _all_dbs = 5000
> > > _dbs_info = 5000
> > > _view = 2500
> > > _view/queries = 2500
> > > _find = 2500
> > > ```
> > >
> > > Latter (after few years) CouchDB would deprecate and remove old
> endpoints.
> > >
> > > Best regards,
> > > iilyak
> > >
> > > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> > >> Hello everyone,
> > >>
> > >> I'd like to discuss the shape and behavior of streaming APIs for
> CouchDB 4.x
> > >>
> > >> By "streaming APIs" I mean APIs which stream data in row as it gets
> > >> read from the database. These are the endpoints I was thinking of:
> > >>
> > >> _all_docs, _all_dbs, _dbs_info and query results
> > >>
> > >> I want to focus on what happens when FoundationDB transactions
> > >> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > >> feeds, will crash or freeze. The reason is because the
> > >> transaction_too_old error at the end of 5 seconds is retry-able by
> > >> default, so the request handlers run again and end up shoving the
> > >> whole request down the socket again, headers and all, which is
> > >> obviously broken and not what we want.
> > >>
> > >> There are few alternatives discussed in couchdb-dev channel. I'll
> > >> present some behaviors but feel free to add more. Some ideas might
> > >> have been discounted on the IRC discussion already but I'll present
> > >> them anyway in case is sparks further conversation:
> > >>
> > >> A) Do what _changes[1] feeds do. Start a new transaction and continue
> > >> streaming the data from the next key after last emitted in the
> > >> previous transaction. Document the API behavior change that it may
> > >> present a view of the data is never a point-in-time[4] snapshot of the
> > >> DB.
> > >>
> > >> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > >> don't have to change to continue using these CouchDB 4.0 endpoints
> > >> - This is the easiest to implement since it would re-use the
> > >> implementation for _changes feed (an extra option passed to the fold
> > >> function).
> > >> - Breaks API behavior if users relied on having a point-in-time[4]
> > >> snapshot view of the data.
> > >>
> > >> B) Simply end the stream. Let the users pass a `?transaction=true`
> > >> param which indicates they are aware the stream may end early and so
> > >> would have to paginate from the last emitted key with a skip=1. This
> > >> will keep the request bodies the same as current CouchDB. However, if
> > >> the users got all the data one request, they will end up wasting
> > >> another request to see if there is more data available. If they didn't
> > >> get any data they might have a too large of a skip value (see [2]) so
> > >> would have to guess different values for start/end keys. Or impose max
> > >> limit for the `skip` parameter.
> > >>
> > >> C) End the stream and add a final metadata row like a "transaction":
> > >> "timeout" at the end. That will let the user know to keep paginating
> > >> from the last key onward. This won't work for `_all_dbs` and
> > >> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > >> feeds and only use this for views and and _all_docs? If we like this
> > >> choice, let's think what happens for those as I couldn't come up with
> > >> anything decent there.
> > >>
> > >> D) Same as C but to solve the issue with skips[2], emit a bookmark
> > >> "key" of where the iteration stopped and the current "skip" and
> > >> "limit" params, which would keep decreasing. Then user would pass
> > >> those in "start_key=..." in the next request along with the limit and
> > >> skip params. So something like "continuation":{"skip":599, "limit":5,
> > >> "key":"..."}. This has the same issue with array results for
> > >> `_all_dbs` and `_dbs_info`[3].
> > >>
> > >> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > >> there such that response time is likely to fit in one transaction.
> > >> This could be tricky as different runtime environments will have
> > >> different characteristics. Also, if the timeout happens there isn't a
> > >> a nice way to send an HTTP error since we already sent the 200
> > >> response. The downside is that this might break how some users use the
> > >> API, if say the are using large skips and limits already. Perhaps here
> > >> we do both B and D, such that if users want transactional behavior,
> > >> they specify that `transaction=true` param and only then we enforce
> > >> low limit and skip maximums.
> > >>
> > >> F) At least for `_all_docs` it seems providing a point-in-time
> > >> snapshot view doesn't necessarily need to be tied to transaction
> > >> boundaries. We could check the update sequence of the database at the
> > >> start of the next transaction and if it hasn't changed we can continue
> > >> emitting a consistent view. This can apply to C and D and would just
> > >> determine when the stream ends. If there are no writes happening to
> > >> the db, this could potential streams all the data just like option A
> > >> would do. Not entirely sure if this would work for views.
> > >>
> > >> So what do we think? I can see different combinations of options here,
> > >> maybe even different for each API point. For example `_all_dbs`,
> > >> `_dbs_info` are always A, and `_all_docs` and views default to A but
> > >> have parameters to do F, etc.
> > >>
> > >> Cheers,
> > >> -Nick
> > >>
> > >> Some footnotes:
> > >>
> > >> [1] _changes feeds is the only one that works currently. It behaves as
> > >> per RFC
> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns
> .
> > >> That is, we continue streaming the data by resetting the transaction
> > >> object and restarting from the last emitted key (db sequence in this
> > >> case). However, because the transaction restarts if a document is
> > >> updated while the streaming take place, it may appear in the _changes
> > >> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > >> have to document it, since previously we presented this point-in-time
> > >> snapshot of the database from when we started streaming.
> > >>
> > >> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > >> currently support efficient offsets for key selectors
> > >> (
> https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging
> )
> > >> we implemented skip by iterating over the data. This means that a skip
> > >> of say 100000 could keep timing out the transaction without yielding
> > >> any data.
> > >>
> > >> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > >> obvious place to insert a last metadata row.
> > >>
> > >> [4] For example they have a constraint that documents "a" and "z"
> > >> cannot both be in the database at the same time. But when iterating
> > >> it's possible that "a" was there at the start. Then by the end, "a"
> > >> was removed and "z" added, so both "a" and "z" would appear in the
> > >> emitted stream. Note that FoundationDB has APIs which exhibit the same
> > >> "relaxed" constrains:
> > >>
> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> > >>
> >
> >
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
- page is to provide some notion of progress for user
- timestamp - I was thinking that we should drop requests if user would try to pass bookmark created an hour ago.
On 2020/04/22 21:58:40, Robert Samuel Newson <rn...@apache.org> wrote:
> "page" and "page number" are odd to me as these don't exist as concepts, I'd rather not invent them. I note there's no mention of page size, which makes "page number" very vague.
>
> What is "timestamp" in the bookmark and what effect does it have when the bookmark is passed back in?
>
> I guess, why does the bookmark include so much extraneous data? Items that are not needed to find the fdb key to begin the next response from.
>
>
> > On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
> >
> > Hello everyone,
> >
> > Based on the discussions on the thread I would like to propose a number of first steps:
> > 1) introduce new endpoints
> > - {db}/_all_docs/page
> > - {db}/_all_docs/queries/page
> > - _all_dbs/page
> > - _dbs_info/page
> > - {db}/_design/{ddoc}/_view/{view}/page
> > - {db}/_design/{ddoc}/_view/{view}/queries/page
> > - {db}/_find/page
> >
> > These new endpoints would act as follows:
> > - don't use delayed responses
> > - return object with following structure
> > ```
> > {
> > "total": Total,
> > "bookmark": base64 encoded opaque value,
> > "completed": true | false,
> > "update_seq": when available,
> > "page": current page number,
> > "items": [
> > ]
> > }
> > ```
> > - the bookmark would include following data (base64 or protobuff???):
> > - direction
> > - page
> > - descending
> > - endkey
> > - endkey_docid
> > - inclusive_end
> > - startkey
> > - startkey_docid
> > - last_key
> > - update_seq
> > - timestamp
> > ```
> >
> > 2) Implement per-endpoint configurable max limits
> > ```
> > _all_docs = 5000
> > _all_docs/queries = 5000
> > _all_dbs = 5000
> > _dbs_info = 5000
> > _view = 2500
> > _view/queries = 2500
> > _find = 2500
> > ```
> >
> > Latter (after few years) CouchDB would deprecate and remove old endpoints.
> >
> > Best regards,
> > iilyak
> >
> > On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> >> Hello everyone,
> >>
> >> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> >>
> >> By "streaming APIs" I mean APIs which stream data in row as it gets
> >> read from the database. These are the endpoints I was thinking of:
> >>
> >> _all_docs, _all_dbs, _dbs_info and query results
> >>
> >> I want to focus on what happens when FoundationDB transactions
> >> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> >> feeds, will crash or freeze. The reason is because the
> >> transaction_too_old error at the end of 5 seconds is retry-able by
> >> default, so the request handlers run again and end up shoving the
> >> whole request down the socket again, headers and all, which is
> >> obviously broken and not what we want.
> >>
> >> There are few alternatives discussed in couchdb-dev channel. I'll
> >> present some behaviors but feel free to add more. Some ideas might
> >> have been discounted on the IRC discussion already but I'll present
> >> them anyway in case is sparks further conversation:
> >>
> >> A) Do what _changes[1] feeds do. Start a new transaction and continue
> >> streaming the data from the next key after last emitted in the
> >> previous transaction. Document the API behavior change that it may
> >> present a view of the data is never a point-in-time[4] snapshot of the
> >> DB.
> >>
> >> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> >> don't have to change to continue using these CouchDB 4.0 endpoints
> >> - This is the easiest to implement since it would re-use the
> >> implementation for _changes feed (an extra option passed to the fold
> >> function).
> >> - Breaks API behavior if users relied on having a point-in-time[4]
> >> snapshot view of the data.
> >>
> >> B) Simply end the stream. Let the users pass a `?transaction=true`
> >> param which indicates they are aware the stream may end early and so
> >> would have to paginate from the last emitted key with a skip=1. This
> >> will keep the request bodies the same as current CouchDB. However, if
> >> the users got all the data one request, they will end up wasting
> >> another request to see if there is more data available. If they didn't
> >> get any data they might have a too large of a skip value (see [2]) so
> >> would have to guess different values for start/end keys. Or impose max
> >> limit for the `skip` parameter.
> >>
> >> C) End the stream and add a final metadata row like a "transaction":
> >> "timeout" at the end. That will let the user know to keep paginating
> >> from the last key onward. This won't work for `_all_dbs` and
> >> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> >> feeds and only use this for views and and _all_docs? If we like this
> >> choice, let's think what happens for those as I couldn't come up with
> >> anything decent there.
> >>
> >> D) Same as C but to solve the issue with skips[2], emit a bookmark
> >> "key" of where the iteration stopped and the current "skip" and
> >> "limit" params, which would keep decreasing. Then user would pass
> >> those in "start_key=..." in the next request along with the limit and
> >> skip params. So something like "continuation":{"skip":599, "limit":5,
> >> "key":"..."}. This has the same issue with array results for
> >> `_all_dbs` and `_dbs_info`[3].
> >>
> >> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> >> there such that response time is likely to fit in one transaction.
> >> This could be tricky as different runtime environments will have
> >> different characteristics. Also, if the timeout happens there isn't a
> >> a nice way to send an HTTP error since we already sent the 200
> >> response. The downside is that this might break how some users use the
> >> API, if say the are using large skips and limits already. Perhaps here
> >> we do both B and D, such that if users want transactional behavior,
> >> they specify that `transaction=true` param and only then we enforce
> >> low limit and skip maximums.
> >>
> >> F) At least for `_all_docs` it seems providing a point-in-time
> >> snapshot view doesn't necessarily need to be tied to transaction
> >> boundaries. We could check the update sequence of the database at the
> >> start of the next transaction and if it hasn't changed we can continue
> >> emitting a consistent view. This can apply to C and D and would just
> >> determine when the stream ends. If there are no writes happening to
> >> the db, this could potential streams all the data just like option A
> >> would do. Not entirely sure if this would work for views.
> >>
> >> So what do we think? I can see different combinations of options here,
> >> maybe even different for each API point. For example `_all_dbs`,
> >> `_dbs_info` are always A, and `_all_docs` and views default to A but
> >> have parameters to do F, etc.
> >>
> >> Cheers,
> >> -Nick
> >>
> >> Some footnotes:
> >>
> >> [1] _changes feeds is the only one that works currently. It behaves as
> >> per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> >> That is, we continue streaming the data by resetting the transaction
> >> object and restarting from the last emitted key (db sequence in this
> >> case). However, because the transaction restarts if a document is
> >> updated while the streaming take place, it may appear in the _changes
> >> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> >> have to document it, since previously we presented this point-in-time
> >> snapshot of the database from when we started streaming.
> >>
> >> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> >> currently support efficient offsets for key selectors
> >> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> >> we implemented skip by iterating over the data. This means that a skip
> >> of say 100000 could keep timing out the transaction without yielding
> >> any data.
> >>
> >> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> >> obvious place to insert a last metadata row.
> >>
> >> [4] For example they have a constraint that documents "a" and "z"
> >> cannot both be in the database at the same time. But when iterating
> >> it's possible that "a" was there at the start. Then by the end, "a"
> >> was removed and "z" added, so both "a" and "z" would appear in the
> >> emitted stream. Note that FoundationDB has APIs which exhibit the same
> >> "relaxed" constrains:
> >> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >>
>
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Robert Samuel Newson <rn...@apache.org>.
"page" and "page number" are odd to me as these don't exist as concepts, I'd rather not invent them. I note there's no mention of page size, which makes "page number" very vague.
What is "timestamp" in the bookmark and what effect does it have when the bookmark is passed back in?
I guess, why does the bookmark include so much extraneous data? Items that are not needed to find the fdb key to begin the next response from.
> On 22 Apr 2020, at 21:18, Ilya Khlopotov <ii...@apache.org> wrote:
>
> Hello everyone,
>
> Based on the discussions on the thread I would like to propose a number of first steps:
> 1) introduce new endpoints
> - {db}/_all_docs/page
> - {db}/_all_docs/queries/page
> - _all_dbs/page
> - _dbs_info/page
> - {db}/_design/{ddoc}/_view/{view}/page
> - {db}/_design/{ddoc}/_view/{view}/queries/page
> - {db}/_find/page
>
> These new endpoints would act as follows:
> - don't use delayed responses
> - return object with following structure
> ```
> {
> "total": Total,
> "bookmark": base64 encoded opaque value,
> "completed": true | false,
> "update_seq": when available,
> "page": current page number,
> "items": [
> ]
> }
> ```
> - the bookmark would include following data (base64 or protobuff???):
> - direction
> - page
> - descending
> - endkey
> - endkey_docid
> - inclusive_end
> - startkey
> - startkey_docid
> - last_key
> - update_seq
> - timestamp
> ```
>
> 2) Implement per-endpoint configurable max limits
> ```
> _all_docs = 5000
> _all_docs/queries = 5000
> _all_dbs = 5000
> _dbs_info = 5000
> _view = 2500
> _view/queries = 2500
> _find = 2500
> ```
>
> Latter (after few years) CouchDB would deprecate and remove old endpoints.
>
> Best regards,
> iilyak
>
> On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
>> Hello everyone,
>>
>> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>>
>> By "streaming APIs" I mean APIs which stream data in row as it gets
>> read from the database. These are the endpoints I was thinking of:
>>
>> _all_docs, _all_dbs, _dbs_info and query results
>>
>> I want to focus on what happens when FoundationDB transactions
>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>> feeds, will crash or freeze. The reason is because the
>> transaction_too_old error at the end of 5 seconds is retry-able by
>> default, so the request handlers run again and end up shoving the
>> whole request down the socket again, headers and all, which is
>> obviously broken and not what we want.
>>
>> There are few alternatives discussed in couchdb-dev channel. I'll
>> present some behaviors but feel free to add more. Some ideas might
>> have been discounted on the IRC discussion already but I'll present
>> them anyway in case is sparks further conversation:
>>
>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>> streaming the data from the next key after last emitted in the
>> previous transaction. Document the API behavior change that it may
>> present a view of the data is never a point-in-time[4] snapshot of the
>> DB.
>>
>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>> don't have to change to continue using these CouchDB 4.0 endpoints
>> - This is the easiest to implement since it would re-use the
>> implementation for _changes feed (an extra option passed to the fold
>> function).
>> - Breaks API behavior if users relied on having a point-in-time[4]
>> snapshot view of the data.
>>
>> B) Simply end the stream. Let the users pass a `?transaction=true`
>> param which indicates they are aware the stream may end early and so
>> would have to paginate from the last emitted key with a skip=1. This
>> will keep the request bodies the same as current CouchDB. However, if
>> the users got all the data one request, they will end up wasting
>> another request to see if there is more data available. If they didn't
>> get any data they might have a too large of a skip value (see [2]) so
>> would have to guess different values for start/end keys. Or impose max
>> limit for the `skip` parameter.
>>
>> C) End the stream and add a final metadata row like a "transaction":
>> "timeout" at the end. That will let the user know to keep paginating
>> from the last key onward. This won't work for `_all_dbs` and
>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>> feeds and only use this for views and and _all_docs? If we like this
>> choice, let's think what happens for those as I couldn't come up with
>> anything decent there.
>>
>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>> "key" of where the iteration stopped and the current "skip" and
>> "limit" params, which would keep decreasing. Then user would pass
>> those in "start_key=..." in the next request along with the limit and
>> skip params. So something like "continuation":{"skip":599, "limit":5,
>> "key":"..."}. This has the same issue with array results for
>> `_all_dbs` and `_dbs_info`[3].
>>
>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>> there such that response time is likely to fit in one transaction.
>> This could be tricky as different runtime environments will have
>> different characteristics. Also, if the timeout happens there isn't a
>> a nice way to send an HTTP error since we already sent the 200
>> response. The downside is that this might break how some users use the
>> API, if say the are using large skips and limits already. Perhaps here
>> we do both B and D, such that if users want transactional behavior,
>> they specify that `transaction=true` param and only then we enforce
>> low limit and skip maximums.
>>
>> F) At least for `_all_docs` it seems providing a point-in-time
>> snapshot view doesn't necessarily need to be tied to transaction
>> boundaries. We could check the update sequence of the database at the
>> start of the next transaction and if it hasn't changed we can continue
>> emitting a consistent view. This can apply to C and D and would just
>> determine when the stream ends. If there are no writes happening to
>> the db, this could potential streams all the data just like option A
>> would do. Not entirely sure if this would work for views.
>>
>> So what do we think? I can see different combinations of options here,
>> maybe even different for each API point. For example `_all_dbs`,
>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>> have parameters to do F, etc.
>>
>> Cheers,
>> -Nick
>>
>> Some footnotes:
>>
>> [1] _changes feeds is the only one that works currently. It behaves as
>> per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
>> That is, we continue streaming the data by resetting the transaction
>> object and restarting from the last emitted key (db sequence in this
>> case). However, because the transaction restarts if a document is
>> updated while the streaming take place, it may appear in the _changes
>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>> have to document it, since previously we presented this point-in-time
>> snapshot of the database from when we started streaming.
>>
>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>> currently support efficient offsets for key selectors
>> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
>> we implemented skip by iterating over the data. This means that a skip
>> of say 100000 could keep timing out the transaction without yielding
>> any data.
>>
>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>> obvious place to insert a last metadata row.
>>
>> [4] For example they have a constraint that documents "a" and "z"
>> cannot both be in the database at the same time. But when iterating
>> it's possible that "a" was there at the start. Then by the end, "a"
>> was removed and "z" added, so both "a" and "z" would appear in the
>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>> "relaxed" constrains:
>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Ilya Khlopotov <ii...@apache.org>.
Hello everyone,
Based on the discussions on the thread I would like to propose a number of first steps:
1) introduce new endpoints
- {db}/_all_docs/page
- {db}/_all_docs/queries/page
- _all_dbs/page
- _dbs_info/page
- {db}/_design/{ddoc}/_view/{view}/page
- {db}/_design/{ddoc}/_view/{view}/queries/page
- {db}/_find/page
These new endpoints would act as follows:
- don't use delayed responses
- return object with following structure
```
{
"total": Total,
"bookmark": base64 encoded opaque value,
"completed": true | false,
"update_seq": when available,
"page": current page number,
"items": [
]
}
```
- the bookmark would include following data (base64 or protobuff???):
- direction
- page
- descending
- endkey
- endkey_docid
- inclusive_end
- startkey
- startkey_docid
- last_key
- update_seq
- timestamp
```
2) Implement per-endpoint configurable max limits
```
_all_docs = 5000
_all_docs/queries = 5000
_all_dbs = 5000
_dbs_info = 5000
_view = 2500
_view/queries = 2500
_find = 2500
```
Latter (after few years) CouchDB would deprecate and remove old endpoints.
Best regards,
iilyak
On 2020/02/19 22:39:45, Nick Vatamaniuc <va...@apache.org> wrote:
> Hello everyone,
>
> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>
> By "streaming APIs" I mean APIs which stream data in row as it gets
> read from the database. These are the endpoints I was thinking of:
>
> _all_docs, _all_dbs, _dbs_info and query results
>
> I want to focus on what happens when FoundationDB transactions
> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> feeds, will crash or freeze. The reason is because the
> transaction_too_old error at the end of 5 seconds is retry-able by
> default, so the request handlers run again and end up shoving the
> whole request down the socket again, headers and all, which is
> obviously broken and not what we want.
>
> There are few alternatives discussed in couchdb-dev channel. I'll
> present some behaviors but feel free to add more. Some ideas might
> have been discounted on the IRC discussion already but I'll present
> them anyway in case is sparks further conversation:
>
> A) Do what _changes[1] feeds do. Start a new transaction and continue
> streaming the data from the next key after last emitted in the
> previous transaction. Document the API behavior change that it may
> present a view of the data is never a point-in-time[4] snapshot of the
> DB.
>
> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> don't have to change to continue using these CouchDB 4.0 endpoints
> - This is the easiest to implement since it would re-use the
> implementation for _changes feed (an extra option passed to the fold
> function).
> - Breaks API behavior if users relied on having a point-in-time[4]
> snapshot view of the data.
>
> B) Simply end the stream. Let the users pass a `?transaction=true`
> param which indicates they are aware the stream may end early and so
> would have to paginate from the last emitted key with a skip=1. This
> will keep the request bodies the same as current CouchDB. However, if
> the users got all the data one request, they will end up wasting
> another request to see if there is more data available. If they didn't
> get any data they might have a too large of a skip value (see [2]) so
> would have to guess different values for start/end keys. Or impose max
> limit for the `skip` parameter.
>
> C) End the stream and add a final metadata row like a "transaction":
> "timeout" at the end. That will let the user know to keep paginating
> from the last key onward. This won't work for `_all_dbs` and
> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> feeds and only use this for views and and _all_docs? If we like this
> choice, let's think what happens for those as I couldn't come up with
> anything decent there.
>
> D) Same as C but to solve the issue with skips[2], emit a bookmark
> "key" of where the iteration stopped and the current "skip" and
> "limit" params, which would keep decreasing. Then user would pass
> those in "start_key=..." in the next request along with the limit and
> skip params. So something like "continuation":{"skip":599, "limit":5,
> "key":"..."}. This has the same issue with array results for
> `_all_dbs` and `_dbs_info`[3].
>
> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> there such that response time is likely to fit in one transaction.
> This could be tricky as different runtime environments will have
> different characteristics. Also, if the timeout happens there isn't a
> a nice way to send an HTTP error since we already sent the 200
> response. The downside is that this might break how some users use the
> API, if say the are using large skips and limits already. Perhaps here
> we do both B and D, such that if users want transactional behavior,
> they specify that `transaction=true` param and only then we enforce
> low limit and skip maximums.
>
> F) At least for `_all_docs` it seems providing a point-in-time
> snapshot view doesn't necessarily need to be tied to transaction
> boundaries. We could check the update sequence of the database at the
> start of the next transaction and if it hasn't changed we can continue
> emitting a consistent view. This can apply to C and D and would just
> determine when the stream ends. If there are no writes happening to
> the db, this could potential streams all the data just like option A
> would do. Not entirely sure if this would work for views.
>
> So what do we think? I can see different combinations of options here,
> maybe even different for each API point. For example `_all_dbs`,
> `_dbs_info` are always A, and `_all_docs` and views default to A but
> have parameters to do F, etc.
>
> Cheers,
> -Nick
>
> Some footnotes:
>
> [1] _changes feeds is the only one that works currently. It behaves as
> per RFC https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> That is, we continue streaming the data by resetting the transaction
> object and restarting from the last emitted key (db sequence in this
> case). However, because the transaction restarts if a document is
> updated while the streaming take place, it may appear in the _changes
> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> have to document it, since previously we presented this point-in-time
> snapshot of the database from when we started streaming.
>
> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> currently support efficient offsets for key selectors
> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> we implemented skip by iterating over the data. This means that a skip
> of say 100000 could keep timing out the transaction without yielding
> any data.
>
> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> obvious place to insert a last metadata row.
>
> [4] For example they have a constraint that documents "a" and "z"
> cannot both be in the database at the same time. But when iterating
> it's possible that "a" was there at the start. Then by the end, "a"
> was removed and "z" added, so both "a" and "z" would appear in the
> emitted stream. Note that FoundationDB has APIs which exhibit the same
> "relaxed" constrains:
> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Gordon Baird <go...@bairdpartnersltd.com>.
I would like to chime in on this (apologies if it is out of sequence or if not right context, but would like to raise the concept). One of the issues we face for our enterprise deployments in considering CouchDB is our ability to log or serialize the edits and changes to the documents (a kind of version control or version back reference ability - we are considering Couch for regulatory compliant installs with strict audit controls and reporting requirements) - - We had been talking internally that it would be a material improvement for CouchDB if it had something like the following…
Allow a database to be configured with an option to… have an external system subscribe to changes to a “document record” that then would allow for the full pre-change document to be sent to a separate application / logging system … at the current point go the “revision” , allow for a “copy” of the original, before change occurred to be made, flagged and if subscribed to, have that full document sent out to a third party system… something similar to how RabbitMQ works with a separate login system “hooked” into the message broker but not integrated so that does not slow down performance or disrupt the native Erlang flow. Understand that this extra step of “copy and send” original prior to “revision update and store” involved an additional “copy” and might impact performance some, maybe this is a configurable option for each database that can be toggled if the audit feature is more valuable than the give up on some performance. Something like this would be very valuable because it would allow a separate application to keep track of the versions of older “json documents” - that were native CouchDB, but without burdening CouchDB to try and be a full versioning, serializable database.. [maybe just send the raw native Erlang message / map as the message ..]
Again, apologies if out of context or not right place to chime in on this. We are still somewhat new to and still learning all the features and nuances in consideration for our enterprise applications. If helpful, can expand on further - both use cases and design / BIF libraries ideas …
Gordon
> On Apr 8, 2020, at 7:25 PM, Nick Vatamaniuc <va...@gmail.com> wrote:
>
> Thanks for replying, Adam!
>
> Thinking about it some more, it seems there are two benefits
> to changing the streaming APIs:
>
> 1) To provide users with a serializable snapshot. We don't currently
> have that, as Mike pointed out, unless we use n=1&q=1 or CouchDB
> version 1.x. It would be nice to get that with a new release.
>
> 2) To avoid the general anti-pattern of streaming all the data in one
> single request or using a very large skip or limit values.
>
> However, I think the two improvements are not necessarily tied to each
> other. For example, we could set configurable mandatory max limits
> (option E) for all the streaming endpoints in 3.x as well. On the
> other hand, even with a single transaction we could stream say 150k
> rows in 5 seconds. If at some future point FDB would allow minute long
> transactions, we could stream millions of rows before timing out and
> it would still not be a desirable pattern of usage. This is also
> basically option F in a read-only case (we can emit a snapshot as long
> as there are no writes to that particular db), and I think we agree
> that it is not that appealing of an option.
>
> What do we think then about having per-endpoint configurable max
> limits (option E). The configuration could look something like:
>
> [request_limits]
> all_docs = 5000
> views = 2500
> list_dbs = 1000
> dbs_info = 500
>
> If those limits are set, and a request is made against an endpoint
> without the limit parameter, or a limit or skip is provided but is
> greater than the maximum, it would return back immediately with an
> error (412) and an indication of what the a max limit value is.
>
> And I agree that client libraries are important in helping here.
> So for example, Cloudant client libraries could detect that error and
> either return it to the user, or, as a compatibility mode, use a
> few consecutive requests behind the scenes to stream all the data back
> to the user as requested without the users application code needing
> any updates at all.
>
> If those limits are not set, the API would behave as it does now. This
> would provide a smoother upgrade path from 3.x to 4.x so users
> wouldn't have to rewrite their applications.
>
> I still think the bookmarking approach is interesting, but I think it
> might work better for a new API endpoint or enabled with an explicit
> parameter. I can see a case where users might be using Python requests
> library to fetch _all_docs, assuming it gets all the data, then after
> the upgrade same API endpoint suddenly returns only a fraction of the
> rows. There might be another "bookmark" field there but it is buried
> under a few application layers and gets ignored. They users just notice
> the missing data at some point, and it could be perceived as data loss
> in a sense.
>
>> how much value do we derive from that streaming behavior if we
> aggressively limit the `limit`?
>
> Oh good point! That makes sense. We might be able to simplify
> quite a bit of logic internally if we didn't actually stream the data.
> We buffer thousands of doc updates for _bulk_docs already so perhaps
> it is not that different doing it when reading data in these APIs as well.
> It is something that we'd have to experiment with and see how it would
> behaves.
>
> -Nick
>
> On Wed, Apr 1, 2020 at 9:07 PM Adam Kocoloski <ko...@apache.org> wrote:
>>
>> This is a really important topic; thanks Nick for bringing it up. Sorry I didn’t comment earlier. I think Mike neatly captures my perspective with this bit:
>>
>>>> Our current behaviour seems extremely subtle and, I'd argue, unexpected. It is hard to reason about if you really need a particular guarantee.
>>>>
>>>> Is there an opportunity to clarify behaviour here, such that we really _do_ guarantee point-in-time within _any_ single request, but only do this by leveraging FoundationDB's transaction isolation semantics and as such are only able to offer this based on the 5s timeout in place? The request boundary offers a very clear cut, user-visible boundary. This would obviously need to cover reads/writes of single docs and so on as well as probably needing further work w.r.t. bulk docs etc.
>>>>
>>>> This restriction may naturally loosen as FoundationDB improves and the 5s timeout may be increased.
>>
>> It’d be great if we could agree on this use of serializable snapshot isolation under the hood for each response to a CouchDB API request (excepting _changes) as an optimal state.
>>
>> Of course, we have this complicating factor of an existing API and a community of users running applications in production against that API :) As you can imagine from the above, I’d be opposed to A); I think that squanders a real opportunity that we have here with a new major version. I also think that the return on investment for F) is too low; a large portion of our production databases see a 24/7 write load so a code path that only activates when a DB is quiesced doesn’t get my vote.
>>
>> When I look at the other options, I think it’s important to take a broader view and consider the user experience in the client libraries as well as the API. Our experience at IBM Cloud is that a large majority of API requests come from a well-defined set of client libraries, and as we consider non-trivial changes to the API we can look to those libraries as a way to smooth over the API breakage, and intelligently surface new capabilities even if the least-disruptive way to introduce them to the API is a bit janky.
>>
>> As a concrete example, I would support an aggressive ceiling on `limit` and `skip` in the 4.0 API, while enhancing popular client libraries as needed to allow users to opt-in to automatic pagination through larger result sets.
>>
>> Nick rightly points out that we don’t have a good way to declare a read version timeout when we’ve already streamed a portion of the result set to the client, which is something we ought consider even if we do apply the restrictions in E). I acknowledge that I may be opening a can of worms, but ... how much value do we derive from that streaming behavior if we aggressively limit the `limit`? We wouldn’t be holding that much data in memory on the CouchDB side, and I don’t think many of our clients are parsing half-completed JSON objects for anything beyond the _changes feed. Something to think about.
>>
>> Cheers, Adam
>>
>>> On Feb 25, 2020, at 2:52 PM, Nick Vatamaniuc <va...@gmail.com> wrote:
>>>
>>> Hi Mike,
>>>
>>> Good point about CouchDB not actually providing point-in-time
>>> snapshots. I missed those cases when thinking about it.
>>>
>>> I wonder if that points to defaulting to option A since it maintains
>>> the API compatibility and doesn't loosen the current constraints
>>> anyway. At least it will un-break the current version of the branch
>>> until we figure out something better. Otherwise it's completely
>>> unusable for dbs with more than 200-300k documents.
>>>
>>> I like the idea of returning a bookmark and a completed/not-completed
>>> flag. That is, it would be option D for _all_docs and map-reduce
>>> views, but instead of the complex continuation object it would be a
>>> base64-encoded, opaque object. Passing a bookmark back in as a
>>> parameter would be exclusive to passing in a start, end, skip, limit,
>>> and direction (forward/reverse) parameters. For _all_dbs, and
>>> _dbs_info where we don't have a place for metadata rows, we might need
>>> a new API endpoint. And maybe that opens the door to expose more
>>> transactional features in the API in general...
>>>
>>> Also, it seems B, C and F have too many corner cases and
>>> inconsistencies so they can probably be discarded, unless someone
>>> disagrees.
>>>
>>> Configurable skips and limit maximums (E) may still be interesting.
>>> Though, they don't necessarily have to be related to transactions, but
>>> can instead be used to ensure streaming APIs are consumed in smaller
>>> chunks.
>>>
>>> Cheers,
>>> -Nick
>>>
>>>
>>>
>>> On Mon, Feb 24, 2020 at 7:26 AM Mike Rhodes <co...@dx13.co.uk> wrote:
>>>>
>>>> Nick,
>>>>
>>>> Thanks for thinking this through, it's certainly subtle and very unclear what is a "good" solution :(
>>>>
>>>> I have a couple of thoughts, firstly about the guarantees we currently offer and then wondering whether there is an opportunity to improve our API by offering a single guarantee across all request types rather than bifurcating guarantees.
>>>>
>>>> ---
>>>>
>>>> The first point is that, by my reasoning, CouchDB 2.x doesn't actually don't offer a point-in-time guarantee of the following sort currently. I read this as your saying Couch does offer this guarantee, apologies if I'm misreading:
>>>>
>>>>> Document the API behavior change that it may
>>>>> present a view of the data is never a point-in-time[4] snapshot of the
>>>>> DB.
>>>> ...
>>>>> [4] For example they have a constraint that documents "a" and "z"
>>>>> cannot both be in the database at the same time. But when iterating
>>>>> it's possible that "a" was there at the start. Then by the end, "a"
>>>>> was removed and "z" added, so both "a" and "z" would appear in the
>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>>>> "relaxed" constrains:
>>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>>
>>>> I don't believe we offer this guarantee because different database shards will respond to the scatter-gather inherent to a single global query type request at different times. This means that, given the following sequence of events:
>>>>
>>>> (1) The shard containing "a" may start returning at time N.
>>>> (2) "a" may be deleted at N+1, but (1) will still be streaming from time N.
>>>> (3) "z" may be written to a second shard at time N+2.
>>>> (4) that second shard may not start returning until time N+3.
>>>>
>>>> By my reasoning, "a" and "z" could thus appear in the same result set in current CouchDB, even if they never actually appear in the primary data at the same time (regardless of latency of shard replicas coming into agreement), voiding [4].
>>>>
>>>> By my reckoning, you have point-in-time across a query request when you are working with a single shard, meaning we do have point in time for two scenarios:
>>>>
>>>> - Partitioned queries.
>>>> - Q=1 databases.
>>>>
>>>> Albeit this guarantee is still talking about the point in time of a single shard's replica rather than all replicas, meaning that further requests may produce different results if the shards are not in agreement. Which can perhaps be fixed by using stable=true.
>>>>
>>>> I _think_ the working here is correct, but I'd welcome corrections in my understanding!
>>>>
>>>> ---
>>>>
>>>> Our current behaviour seems extremely subtle and, I'd argue, unexpected. It is hard to reason about if you really need a particular guarantee.
>>>>
>>>> Is there an opportunity to clarify behaviour here, such that we really _do_ guarantee point-in-time within _any_ single request, but only do this by leveraging FoundationDB's transaction isolation semantics and as such are only able to offer this based on the 5s timeout in place? The request boundary offers a very clear cut, user-visible boundary. This would obviously need to cover reads/writes of single docs and so on as well as probably needing further work w.r.t. bulk docs etc.
>>>>
>>>> This restriction may naturally loosen as FoundationDB improves and the 5s timeout may be increased.
>>>>
>>>> In this approach, my preference would be to add a closing line to the result stream to contain both a bookmark (based on the FoundationDB key perhaps rather than the index key of itself to avoid problems with skip/limit?) and a complete/not-complete boolean to enable clients to avoid the extra HTTP round-trip for completed result sets that Nick mentions.
>>>>
>>>> ---
>>>>
>>>> For option (F), I feel that the "it sometimes works and sometimes doesn't" effect of checking the update-seq to see if we can continue streaming will be a confusing experience. I also find something similar with option (A) where a single request covers potentially many points in time and so feels hard to reason about, although it's a bit less subtle than today.
>>>>
>>>> Footnote [2] seems quite a major problem, however, with the single transaction approach and as Nick says, it is hard to pick "good" maximums for skip -- perhaps users need to just avoid use of these in the new system given its behaviour? It feels like there's a definite "against the grain" aspect to these.
>>>>
>>>> --
>>>> Mike.
>>>>
>>>> On Wed, 19 Feb 2020, at 22:39, Nick Vatamaniuc wrote:
>>>>> Hello everyone,
>>>>>
>>>>> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>>>>>
>>>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>>>> read from the database. These are the endpoints I was thinking of:
>>>>>
>>>>> _all_docs, _all_dbs, _dbs_info and query results
>>>>>
>>>>> I want to focus on what happens when FoundationDB transactions
>>>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>>>> feeds, will crash or freeze. The reason is because the
>>>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>>>> default, so the request handlers run again and end up shoving the
>>>>> whole request down the socket again, headers and all, which is
>>>>> obviously broken and not what we want.
>>>>>
>>>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>>>> present some behaviors but feel free to add more. Some ideas might
>>>>> have been discounted on the IRC discussion already but I'll present
>>>>> them anyway in case is sparks further conversation:
>>>>>
>>>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>>>> streaming the data from the next key after last emitted in the
>>>>> previous transaction. Document the API behavior change that it may
>>>>> present a view of the data is never a point-in-time[4] snapshot of the
>>>>> DB.
>>>>>
>>>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>>>> - This is the easiest to implement since it would re-use the
>>>>> implementation for _changes feed (an extra option passed to the fold
>>>>> function).
>>>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>>>> snapshot view of the data.
>>>>>
>>>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>>>> param which indicates they are aware the stream may end early and so
>>>>> would have to paginate from the last emitted key with a skip=1. This
>>>>> will keep the request bodies the same as current CouchDB. However, if
>>>>> the users got all the data one request, they will end up wasting
>>>>> another request to see if there is more data available. If they didn't
>>>>> get any data they might have a too large of a skip value (see [2]) so
>>>>> would have to guess different values for start/end keys. Or impose max
>>>>> limit for the `skip` parameter.
>>>>>
>>>>> C) End the stream and add a final metadata row like a "transaction":
>>>>> "timeout" at the end. That will let the user know to keep paginating
>>>>> from the last key onward. This won't work for `_all_dbs` and
>>>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>>>> feeds and only use this for views and and _all_docs? If we like this
>>>>> choice, let's think what happens for those as I couldn't come up with
>>>>> anything decent there.
>>>>>
>>>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>>>> "key" of where the iteration stopped and the current "skip" and
>>>>> "limit" params, which would keep decreasing. Then user would pass
>>>>> those in "start_key=..." in the next request along with the limit and
>>>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>>>> "key":"..."}. This has the same issue with array results for
>>>>> `_all_dbs` and `_dbs_info`[3].
>>>>>
>>>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>>>> there such that response time is likely to fit in one transaction.
>>>>> This could be tricky as different runtime environments will have
>>>>> different characteristics. Also, if the timeout happens there isn't a
>>>>> a nice way to send an HTTP error since we already sent the 200
>>>>> response. The downside is that this might break how some users use the
>>>>> API, if say the are using large skips and limits already. Perhaps here
>>>>> we do both B and D, such that if users want transactional behavior,
>>>>> they specify that `transaction=true` param and only then we enforce
>>>>> low limit and skip maximums.
>>>>>
>>>>> F) At least for `_all_docs` it seems providing a point-in-time
>>>>> snapshot view doesn't necessarily need to be tied to transaction
>>>>> boundaries. We could check the update sequence of the database at the
>>>>> start of the next transaction and if it hasn't changed we can continue
>>>>> emitting a consistent view. This can apply to C and D and would just
>>>>> determine when the stream ends. If there are no writes happening to
>>>>> the db, this could potential streams all the data just like option A
>>>>> would do. Not entirely sure if this would work for views.
>>>>>
>>>>> So what do we think? I can see different combinations of options here,
>>>>> maybe even different for each API point. For example `_all_dbs`,
>>>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>>>> have parameters to do F, etc.
>>>>>
>>>>> Cheers,
>>>>> -Nick
>>>>>
>>>>> Some footnotes:
>>>>>
>>>>> [1] _changes feeds is the only one that works currently. It behaves as
>>>>> per RFC
>>>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
>>>>> That is, we continue streaming the data by resetting the transaction
>>>>> object and restarting from the last emitted key (db sequence in this
>>>>> case). However, because the transaction restarts if a document is
>>>>> updated while the streaming take place, it may appear in the _changes
>>>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>>>> have to document it, since previously we presented this point-in-time
>>>>> snapshot of the database from when we started streaming.
>>>>>
>>>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>>>> currently support efficient offsets for key selectors
>>>>> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
>>>>> we implemented skip by iterating over the data. This means that a skip
>>>>> of say 100000 could keep timing out the transaction without yielding
>>>>> any data.
>>>>>
>>>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>>>> obvious place to insert a last metadata row.
>>>>>
>>>>> [4] For example they have a constraint that documents "a" and "z"
>>>>> cannot both be in the database at the same time. But when iterating
>>>>> it's possible that "a" was there at the start. Then by the end, "a"
>>>>> was removed and "z" added, so both "a" and "z" would appear in the
>>>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>>>> "relaxed" constrains:
>>>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>>>
>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Nick Vatamaniuc <va...@gmail.com>.
Hi Rich,
Thanks for contributing!
Regarding the 412, you're right, I mainly went by the 412 being
returned if a database exists and we try to create one [1]. The
precondition there is in the start line of the http request, just like
`...?limit=...` would be. However a 400 would work better in general
so I agree.
In principle I do like the bookmarks better but it also seems like a
large change to the API so am on the fence there. Also interestingly
it would almost be easier to change _list_dbs, _dbs_info and other
endpoints which return plain arrays to return {"total": ..., "items":
[...], ...} objects since breakage would be really obvious, in other
words old client will fail right away as opposed returning misleading
data.
Cheers,
-Nick
[1] https://docs.couchdb.org/en/stable/api/database/common.html#put--db
On Thu, Apr 9, 2020 at 12:10 PM Richard Ellis <RI...@uk.ibm.com> wrote:
>
> HI Nick,
>
> I think that if client side code is expected to make multiple requests
> then those requests should be made as easy as possible. So whilst a client
> library can implement complicated pagination recipes (like the current
> Couch view one) - it is much simpler to collect and send a single
> bookmark/token. Especially so if the naming and structural position of the
> bookmark in requests and responses is consistent across all endpoints
> supporting pagination such that the client side code for pagination is
> easily reusable. I'm in favour of anything supporting pagination to
> provide a bookmark/token based system.
>
> Also if there are limits applied to limits then I'd expect that anything
> out of that accepted range would be a 400 bad request - IIUC 412
> Precondition failed has specific meaning relating to matching headers
> https://tools.ietf.org/html/rfc2616#section-10.4.13 which I don't think
> apply in this case.
>
> Rich
>
>
>
> From: Nick Vatamaniuc <va...@gmail.com>
> To: dev@couchdb.apache.org
> Date: 09/04/2020 00:25
> Subject: [EXTERNAL] Re: [DISCUSS] Streaming API in CouchDB 4.0
>
>
>
> Thanks for replying, Adam!
>
> Thinking about it some more, it seems there are two benefits
> to changing the streaming APIs:
>
> 1) To provide users with a serializable snapshot. We don't currently
> have that, as Mike pointed out, unless we use n=1&q=1 or CouchDB
> version 1.x. It would be nice to get that with a new release.
>
> 2) To avoid the general anti-pattern of streaming all the data in one
> single request or using a very large skip or limit values.
>
> However, I think the two improvements are not necessarily tied to each
> other. For example, we could set configurable mandatory max limits
> (option E) for all the streaming endpoints in 3.x as well. On the
> other hand, even with a single transaction we could stream say 150k
> rows in 5 seconds. If at some future point FDB would allow minute long
> transactions, we could stream millions of rows before timing out and
> it would still not be a desirable pattern of usage. This is also
> basically option F in a read-only case (we can emit a snapshot as long
> as there are no writes to that particular db), and I think we agree
> that it is not that appealing of an option.
>
> What do we think then about having per-endpoint configurable max
> limits (option E). The configuration could look something like:
>
> [request_limits]
> all_docs = 5000
> views = 2500
> list_dbs = 1000
> dbs_info = 500
>
> If those limits are set, and a request is made against an endpoint
> without the limit parameter, or a limit or skip is provided but is
> greater than the maximum, it would return back immediately with an
> error (412) and an indication of what the a max limit value is.
>
> And I agree that client libraries are important in helping here.
> So for example, Cloudant client libraries could detect that error and
> either return it to the user, or, as a compatibility mode, use a
> few consecutive requests behind the scenes to stream all the data back
> to the user as requested without the users application code needing
> any updates at all.
>
> If those limits are not set, the API would behave as it does now. This
> would provide a smoother upgrade path from 3.x to 4.x so users
> wouldn't have to rewrite their applications.
>
> I still think the bookmarking approach is interesting, but I think it
> might work better for a new API endpoint or enabled with an explicit
> parameter. I can see a case where users might be using Python requests
> library to fetch _all_docs, assuming it gets all the data, then after
> the upgrade same API endpoint suddenly returns only a fraction of the
> rows. There might be another "bookmark" field there but it is buried
> under a few application layers and gets ignored. They users just notice
> the missing data at some point, and it could be perceived as data loss
> in a sense.
>
> > how much value do we derive from that streaming behavior if we
> aggressively limit the `limit`?
>
> Oh good point! That makes sense. We might be able to simplify
> quite a bit of logic internally if we didn't actually stream the data.
> We buffer thousands of doc updates for _bulk_docs already so perhaps
> it is not that different doing it when reading data in these APIs as
> well.
> It is something that we'd have to experiment with and see how it would
> behaves.
>
> -Nick
>
> On Wed, Apr 1, 2020 at 9:07 PM Adam Kocoloski <ko...@apache.org> wrote:
> >
> > This is a really important topic; thanks Nick for bringing it up. Sorry
> I didn’t comment earlier. I think Mike neatly captures my perspective with
> this bit:
> >
> > >> Our current behaviour seems extremely subtle and, I'd argue,
> unexpected. It is hard to reason about if you really need a particular
> guarantee.
> > >>
> > >> Is there an opportunity to clarify behaviour here, such that we
> really _do_ guarantee point-in-time within _any_ single request, but only
> do this by leveraging FoundationDB's transaction isolation semantics and
> as such are only able to offer this based on the 5s timeout in place? The
> request boundary offers a very clear cut, user-visible boundary. This
> would obviously need to cover reads/writes of single docs and so on as
> well as probably needing further work w.r.t. bulk docs etc.
> > >>
> > >> This restriction may naturally loosen as FoundationDB improves and
> the 5s timeout may be increased.
> >
> > It’d be great if we could agree on this use of serializable snapshot
> isolation under the hood for each response to a CouchDB API request
> (excepting _changes) as an optimal state.
> >
> > Of course, we have this complicating factor of an existing API and a
> community of users running applications in production against that API :)
> As you can imagine from the above, I’d be opposed to A); I think that
> squanders a real opportunity that we have here with a new major version. I
> also think that the return on investment for F) is too low; a large
> portion of our production databases see a 24/7 write load so a code path
> that only activates when a DB is quiesced doesn’t get my vote.
> >
> > When I look at the other options, I think it’s important to take a
> broader view and consider the user experience in the client libraries as
> well as the API. Our experience at IBM Cloud is that a large majority of
> API requests come from a well-defined set of client libraries, and as we
> consider non-trivial changes to the API we can look to those libraries as
> a way to smooth over the API breakage, and intelligently surface new
> capabilities even if the least-disruptive way to introduce them to the API
> is a bit janky.
> >
> > As a concrete example, I would support an aggressive ceiling on `limit`
> and `skip` in the 4.0 API, while enhancing popular client libraries as
> needed to allow users to opt-in to automatic pagination through larger
> result sets.
> >
> > Nick rightly points out that we don’t have a good way to declare a read
> version timeout when we’ve already streamed a portion of the result set to
> the client, which is something we ought consider even if we do apply the
> restrictions in E). I acknowledge that I may be opening a can of worms,
> but ... how much value do we derive from that streaming behavior if we
> aggressively limit the `limit`? We wouldn’t be holding that much data in
> memory on the CouchDB side, and I don’t think many of our clients are
> parsing half-completed JSON objects for anything beyond the _changes feed.
> Something to think about.
> >
> > Cheers, Adam
> >
> > > On Feb 25, 2020, at 2:52 PM, Nick Vatamaniuc <va...@gmail.com>
> wrote:
> > >
> > > Hi Mike,
> > >
> > > Good point about CouchDB not actually providing point-in-time
> > > snapshots. I missed those cases when thinking about it.
> > >
> > > I wonder if that points to defaulting to option A since it maintains
> > > the API compatibility and doesn't loosen the current constraints
> > > anyway. At least it will un-break the current version of the branch
> > > until we figure out something better. Otherwise it's completely
> > > unusable for dbs with more than 200-300k documents.
> > >
> > > I like the idea of returning a bookmark and a completed/not-completed
> > > flag. That is, it would be option D for _all_docs and map-reduce
> > > views, but instead of the complex continuation object it would be a
> > > base64-encoded, opaque object. Passing a bookmark back in as a
> > > parameter would be exclusive to passing in a start, end, skip, limit,
> > > and direction (forward/reverse) parameters. For _all_dbs, and
> > > _dbs_info where we don't have a place for metadata rows, we might need
> > > a new API endpoint. And maybe that opens the door to expose more
> > > transactional features in the API in general...
> > >
> > > Also, it seems B, C and F have too many corner cases and
> > > inconsistencies so they can probably be discarded, unless someone
> > > disagrees.
> > >
> > > Configurable skips and limit maximums (E) may still be interesting.
> > > Though, they don't necessarily have to be related to transactions, but
> > > can instead be used to ensure streaming APIs are consumed in smaller
> > > chunks.
> > >
> > > Cheers,
> > > -Nick
> > >
> > >
> > >
> > > On Mon, Feb 24, 2020 at 7:26 AM Mike Rhodes <co...@dx13.co.uk>
> wrote:
> > >>
> > >> Nick,
> > >>
> > >> Thanks for thinking this through, it's certainly subtle and very
> unclear what is a "good" solution :(
> > >>
> > >> I have a couple of thoughts, firstly about the guarantees we
> currently offer and then wondering whether there is an opportunity to
> improve our API by offering a single guarantee across all request types
> rather than bifurcating guarantees.
> > >>
> > >> ---
> > >>
> > >> The first point is that, by my reasoning, CouchDB 2.x doesn't
> actually don't offer a point-in-time guarantee of the following sort
> currently. I read this as your saying Couch does offer this guarantee,
> apologies if I'm misreading:
> > >>
> > >>> Document the API behavior change that it may
> > >>> present a view of the data is never a point-in-time[4] snapshot of
> the
> > >>> DB.
> > >> ...
> > >>> [4] For example they have a constraint that documents "a" and "z"
> > >>> cannot both be in the database at the same time. But when iterating
> > >>> it's possible that "a" was there at the start. Then by the end, "a"
> > >>> was removed and "z" added, so both "a" and "z" would appear in the
> > >>> emitted stream. Note that FoundationDB has APIs which exhibit the
> same
> > >>> "relaxed" constrains:
> > >>>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__apple.github.io_foundationdb_api-2Dpython.html-23module-2Dfdb.locality&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=2FAZh1XhUm-BtJzY8Lca-ZC-76K6zLu2Q7zpFojCGk8&s=HzP2Pj3x5vl5LP656xQI81QM2YyZZuPN1HYbw_L7jzE&e=
>
> > >>
> > >> I don't believe we offer this guarantee because different database
> shards will respond to the scatter-gather inherent to a single global
> query type request at different times. This means that, given the
> following sequence of events:
> > >>
> > >> (1) The shard containing "a" may start returning at time N.
> > >> (2) "a" may be deleted at N+1, but (1) will still be streaming from
> time N.
> > >> (3) "z" may be written to a second shard at time N+2.
> > >> (4) that second shard may not start returning until time N+3.
> > >>
> > >> By my reasoning, "a" and "z" could thus appear in the same result set
> in current CouchDB, even if they never actually appear in the primary data
> at the same time (regardless of latency of shard replicas coming into
> agreement), voiding [4].
> > >>
> > >> By my reckoning, you have point-in-time across a query request when
> you are working with a single shard, meaning we do have point in time for
> two scenarios:
> > >>
> > >> - Partitioned queries.
> > >> - Q=1 databases.
> > >>
> > >> Albeit this guarantee is still talking about the point in time of a
> single shard's replica rather than all replicas, meaning that further
> requests may produce different results if the shards are not in agreement.
> Which can perhaps be fixed by using stable=true.
> > >>
> > >> I _think_ the working here is correct, but I'd welcome corrections in
> my understanding!
> > >>
> > >> ---
> > >>
> > >> Our current behaviour seems extremely subtle and, I'd argue,
> unexpected. It is hard to reason about if you really need a particular
> guarantee.
> > >>
> > >> Is there an opportunity to clarify behaviour here, such that we
> really _do_ guarantee point-in-time within _any_ single request, but only
> do this by leveraging FoundationDB's transaction isolation semantics and
> as such are only able to offer this based on the 5s timeout in place? The
> request boundary offers a very clear cut, user-visible boundary. This
> would obviously need to cover reads/writes of single docs and so on as
> well as probably needing further work w.r.t. bulk docs etc.
> > >>
> > >> This restriction may naturally loosen as FoundationDB improves and
> the 5s timeout may be increased.
> > >>
> > >> In this approach, my preference would be to add a closing line to the
> result stream to contain both a bookmark (based on the FoundationDB key
> perhaps rather than the index key of itself to avoid problems with
> skip/limit?) and a complete/not-complete boolean to enable clients to
> avoid the extra HTTP round-trip for completed result sets that Nick
> mentions.
> > >>
> > >> ---
> > >>
> > >> For option (F), I feel that the "it sometimes works and sometimes
> doesn't" effect of checking the update-seq to see if we can continue
> streaming will be a confusing experience. I also find something similar
> with option (A) where a single request covers potentially many points in
> time and so feels hard to reason about, although it's a bit less subtle
> than today.
> > >>
> > >> Footnote [2] seems quite a major problem, however, with the single
> transaction approach and as Nick says, it is hard to pick "good" maximums
> for skip -- perhaps users need to just avoid use of these in the new
> system given its behaviour? It feels like there's a definite "against the
> grain" aspect to these.
> > >>
> > >> --
> > >> Mike.
> > >>
> > >> On Wed, 19 Feb 2020, at 22:39, Nick Vatamaniuc wrote:
> > >>> Hello everyone,
> > >>>
> > >>> I'd like to discuss the shape and behavior of streaming APIs for
> CouchDB 4.x
> > >>>
> > >>> By "streaming APIs" I mean APIs which stream data in row as it gets
> > >>> read from the database. These are the endpoints I was thinking of:
> > >>>
> > >>> _all_docs, _all_dbs, _dbs_info and query results
> > >>>
> > >>> I want to focus on what happens when FoundationDB transactions
> > >>> time-out after 5 seconds. Currently, all those APIs except
> _changes[1]
> > >>> feeds, will crash or freeze. The reason is because the
> > >>> transaction_too_old error at the end of 5 seconds is retry-able by
> > >>> default, so the request handlers run again and end up shoving the
> > >>> whole request down the socket again, headers and all, which is
> > >>> obviously broken and not what we want.
> > >>>
> > >>> There are few alternatives discussed in couchdb-dev channel. I'll
> > >>> present some behaviors but feel free to add more. Some ideas might
> > >>> have been discounted on the IRC discussion already but I'll present
> > >>> them anyway in case is sparks further conversation:
> > >>>
> > >>> A) Do what _changes[1] feeds do. Start a new transaction and
> continue
> > >>> streaming the data from the next key after last emitted in the
> > >>> previous transaction. Document the API behavior change that it may
> > >>> present a view of the data is never a point-in-time[4] snapshot of
> the
> > >>> DB.
> > >>>
> > >>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > >>> don't have to change to continue using these CouchDB 4.0 endpoints
> > >>> - This is the easiest to implement since it would re-use the
> > >>> implementation for _changes feed (an extra option passed to the fold
> > >>> function).
> > >>> - Breaks API behavior if users relied on having a point-in-time[4]
> > >>> snapshot view of the data.
> > >>>
> > >>> B) Simply end the stream. Let the users pass a `?transaction=true`
> > >>> param which indicates they are aware the stream may end early and so
> > >>> would have to paginate from the last emitted key with a skip=1. This
> > >>> will keep the request bodies the same as current CouchDB. However,
> if
> > >>> the users got all the data one request, they will end up wasting
> > >>> another request to see if there is more data available. If they
> didn't
> > >>> get any data they might have a too large of a skip value (see [2])
> so
> > >>> would have to guess different values for start/end keys. Or impose
> max
> > >>> limit for the `skip` parameter.
> > >>>
> > >>> C) End the stream and add a final metadata row like a "transaction":
> > >>> "timeout" at the end. That will let the user know to keep paginating
> > >>> from the last key onward. This won't work for `_all_dbs` and
> > >>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > >>> feeds and only use this for views and and _all_docs? If we like this
> > >>> choice, let's think what happens for those as I couldn't come up
> with
> > >>> anything decent there.
> > >>>
> > >>> D) Same as C but to solve the issue with skips[2], emit a bookmark
> > >>> "key" of where the iteration stopped and the current "skip" and
> > >>> "limit" params, which would keep decreasing. Then user would pass
> > >>> those in "start_key=..." in the next request along with the limit
> and
> > >>> skip params. So something like "continuation":{"skip":599,
> "limit":5,
> > >>> "key":"..."}. This has the same issue with array results for
> > >>> `_all_dbs` and `_dbs_info`[3].
> > >>>
> > >>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > >>> there such that response time is likely to fit in one transaction.
> > >>> This could be tricky as different runtime environments will have
> > >>> different characteristics. Also, if the timeout happens there isn't
> a
> > >>> a nice way to send an HTTP error since we already sent the 200
> > >>> response. The downside is that this might break how some users use
> the
> > >>> API, if say the are using large skips and limits already. Perhaps
> here
> > >>> we do both B and D, such that if users want transactional behavior,
> > >>> they specify that `transaction=true` param and only then we enforce
> > >>> low limit and skip maximums.
> > >>>
> > >>> F) At least for `_all_docs` it seems providing a point-in-time
> > >>> snapshot view doesn't necessarily need to be tied to transaction
> > >>> boundaries. We could check the update sequence of the database at
> the
> > >>> start of the next transaction and if it hasn't changed we can
> continue
> > >>> emitting a consistent view. This can apply to C and D and would just
> > >>> determine when the stream ends. If there are no writes happening to
> > >>> the db, this could potential streams all the data just like option A
> > >>> would do. Not entirely sure if this would work for views.
> > >>>
> > >>> So what do we think? I can see different combinations of options
> here,
> > >>> maybe even different for each API point. For example `_all_dbs`,
> > >>> `_dbs_info` are always A, and `_all_docs` and views default to A but
> > >>> have parameters to do F, etc.
> > >>>
> > >>> Cheers,
> > >>> -Nick
> > >>>
> > >>> Some footnotes:
> > >>>
> > >>> [1] _changes feeds is the only one that works currently. It behaves
> as
> > >>> per RFC
> > >>>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb-2Ddocumentation_blob_master_rfcs_003-2Dfdb-2Dseq-2Dindex.md-23access-2Dpatterns&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=2FAZh1XhUm-BtJzY8Lca-ZC-76K6zLu2Q7zpFojCGk8&s=7Khj1Mm0BvTQkebpPs4O7kE2moJMmmGjV7_icfRo_q8&e=
> .
> > >>> That is, we continue streaming the data by resetting the transaction
> > >>> object and restarting from the last emitted key (db sequence in this
> > >>> case). However, because the transaction restarts if a document is
> > >>> updated while the streaming take place, it may appear in the
> _changes
> > >>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > >>> have to document it, since previously we presented this
> point-in-time
> > >>> snapshot of the database from when we started streaming.
> > >>>
> > >>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > >>> currently support efficient offsets for key selectors
> > >>> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__apple.github.io_foundationdb_known-2Dlimitations.html-23dont-2Duse-2Dkey-2Dselectors-2Dfor-2Dpaging&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=2FAZh1XhUm-BtJzY8Lca-ZC-76K6zLu2Q7zpFojCGk8&s=iJJQDYHRGZ6PQ_FF3sy9nBSygRYkEh3cRPMFYg2Tkq8&e=
> )
> > >>> we implemented skip by iterating over the data. This means that a
> skip
> > >>> of say 100000 could keep timing out the transaction without yielding
> > >>> any data.
> > >>>
> > >>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > >>> obvious place to insert a last metadata row.
> > >>>
> > >>> [4] For example they have a constraint that documents "a" and "z"
> > >>> cannot both be in the database at the same time. But when iterating
> > >>> it's possible that "a" was there at the start. Then by the end, "a"
> > >>> was removed and "z" added, so both "a" and "z" would appear in the
> > >>> emitted stream. Note that FoundationDB has APIs which exhibit the
> same
> > >>> "relaxed" constrains:
> > >>>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__apple.github.io_foundationdb_api-2Dpython.html-23module-2Dfdb.locality&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=2FAZh1XhUm-BtJzY8Lca-ZC-76K6zLu2Q7zpFojCGk8&s=HzP2Pj3x5vl5LP656xQI81QM2YyZZuPN1HYbw_L7jzE&e=
>
> > >>>
> >
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
RE: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Richard Ellis <RI...@uk.ibm.com>.
HI Nick,
I think that if client side code is expected to make multiple requests
then those requests should be made as easy as possible. So whilst a client
library can implement complicated pagination recipes (like the current
Couch view one) - it is much simpler to collect and send a single
bookmark/token. Especially so if the naming and structural position of the
bookmark in requests and responses is consistent across all endpoints
supporting pagination such that the client side code for pagination is
easily reusable. I'm in favour of anything supporting pagination to
provide a bookmark/token based system.
Also if there are limits applied to limits then I'd expect that anything
out of that accepted range would be a 400 bad request - IIUC 412
Precondition failed has specific meaning relating to matching headers
https://tools.ietf.org/html/rfc2616#section-10.4.13 which I don't think
apply in this case.
Rich
From: Nick Vatamaniuc <va...@gmail.com>
To: dev@couchdb.apache.org
Date: 09/04/2020 00:25
Subject: [EXTERNAL] Re: [DISCUSS] Streaming API in CouchDB 4.0
Thanks for replying, Adam!
Thinking about it some more, it seems there are two benefits
to changing the streaming APIs:
1) To provide users with a serializable snapshot. We don't currently
have that, as Mike pointed out, unless we use n=1&q=1 or CouchDB
version 1.x. It would be nice to get that with a new release.
2) To avoid the general anti-pattern of streaming all the data in one
single request or using a very large skip or limit values.
However, I think the two improvements are not necessarily tied to each
other. For example, we could set configurable mandatory max limits
(option E) for all the streaming endpoints in 3.x as well. On the
other hand, even with a single transaction we could stream say 150k
rows in 5 seconds. If at some future point FDB would allow minute long
transactions, we could stream millions of rows before timing out and
it would still not be a desirable pattern of usage. This is also
basically option F in a read-only case (we can emit a snapshot as long
as there are no writes to that particular db), and I think we agree
that it is not that appealing of an option.
What do we think then about having per-endpoint configurable max
limits (option E). The configuration could look something like:
[request_limits]
all_docs = 5000
views = 2500
list_dbs = 1000
dbs_info = 500
If those limits are set, and a request is made against an endpoint
without the limit parameter, or a limit or skip is provided but is
greater than the maximum, it would return back immediately with an
error (412) and an indication of what the a max limit value is.
And I agree that client libraries are important in helping here.
So for example, Cloudant client libraries could detect that error and
either return it to the user, or, as a compatibility mode, use a
few consecutive requests behind the scenes to stream all the data back
to the user as requested without the users application code needing
any updates at all.
If those limits are not set, the API would behave as it does now. This
would provide a smoother upgrade path from 3.x to 4.x so users
wouldn't have to rewrite their applications.
I still think the bookmarking approach is interesting, but I think it
might work better for a new API endpoint or enabled with an explicit
parameter. I can see a case where users might be using Python requests
library to fetch _all_docs, assuming it gets all the data, then after
the upgrade same API endpoint suddenly returns only a fraction of the
rows. There might be another "bookmark" field there but it is buried
under a few application layers and gets ignored. They users just notice
the missing data at some point, and it could be perceived as data loss
in a sense.
> how much value do we derive from that streaming behavior if we
aggressively limit the `limit`?
Oh good point! That makes sense. We might be able to simplify
quite a bit of logic internally if we didn't actually stream the data.
We buffer thousands of doc updates for _bulk_docs already so perhaps
it is not that different doing it when reading data in these APIs as
well.
It is something that we'd have to experiment with and see how it would
behaves.
-Nick
On Wed, Apr 1, 2020 at 9:07 PM Adam Kocoloski <ko...@apache.org> wrote:
>
> This is a really important topic; thanks Nick for bringing it up. Sorry
I didn’t comment earlier. I think Mike neatly captures my perspective with
this bit:
>
> >> Our current behaviour seems extremely subtle and, I'd argue,
unexpected. It is hard to reason about if you really need a particular
guarantee.
> >>
> >> Is there an opportunity to clarify behaviour here, such that we
really _do_ guarantee point-in-time within _any_ single request, but only
do this by leveraging FoundationDB's transaction isolation semantics and
as such are only able to offer this based on the 5s timeout in place? The
request boundary offers a very clear cut, user-visible boundary. This
would obviously need to cover reads/writes of single docs and so on as
well as probably needing further work w.r.t. bulk docs etc.
> >>
> >> This restriction may naturally loosen as FoundationDB improves and
the 5s timeout may be increased.
>
> It’d be great if we could agree on this use of serializable snapshot
isolation under the hood for each response to a CouchDB API request
(excepting _changes) as an optimal state.
>
> Of course, we have this complicating factor of an existing API and a
community of users running applications in production against that API :)
As you can imagine from the above, I’d be opposed to A); I think that
squanders a real opportunity that we have here with a new major version. I
also think that the return on investment for F) is too low; a large
portion of our production databases see a 24/7 write load so a code path
that only activates when a DB is quiesced doesn’t get my vote.
>
> When I look at the other options, I think it’s important to take a
broader view and consider the user experience in the client libraries as
well as the API. Our experience at IBM Cloud is that a large majority of
API requests come from a well-defined set of client libraries, and as we
consider non-trivial changes to the API we can look to those libraries as
a way to smooth over the API breakage, and intelligently surface new
capabilities even if the least-disruptive way to introduce them to the API
is a bit janky.
>
> As a concrete example, I would support an aggressive ceiling on `limit`
and `skip` in the 4.0 API, while enhancing popular client libraries as
needed to allow users to opt-in to automatic pagination through larger
result sets.
>
> Nick rightly points out that we don’t have a good way to declare a read
version timeout when we’ve already streamed a portion of the result set to
the client, which is something we ought consider even if we do apply the
restrictions in E). I acknowledge that I may be opening a can of worms,
but ... how much value do we derive from that streaming behavior if we
aggressively limit the `limit`? We wouldn’t be holding that much data in
memory on the CouchDB side, and I don’t think many of our clients are
parsing half-completed JSON objects for anything beyond the _changes feed.
Something to think about.
>
> Cheers, Adam
>
> > On Feb 25, 2020, at 2:52 PM, Nick Vatamaniuc <va...@gmail.com>
wrote:
> >
> > Hi Mike,
> >
> > Good point about CouchDB not actually providing point-in-time
> > snapshots. I missed those cases when thinking about it.
> >
> > I wonder if that points to defaulting to option A since it maintains
> > the API compatibility and doesn't loosen the current constraints
> > anyway. At least it will un-break the current version of the branch
> > until we figure out something better. Otherwise it's completely
> > unusable for dbs with more than 200-300k documents.
> >
> > I like the idea of returning a bookmark and a completed/not-completed
> > flag. That is, it would be option D for _all_docs and map-reduce
> > views, but instead of the complex continuation object it would be a
> > base64-encoded, opaque object. Passing a bookmark back in as a
> > parameter would be exclusive to passing in a start, end, skip, limit,
> > and direction (forward/reverse) parameters. For _all_dbs, and
> > _dbs_info where we don't have a place for metadata rows, we might need
> > a new API endpoint. And maybe that opens the door to expose more
> > transactional features in the API in general...
> >
> > Also, it seems B, C and F have too many corner cases and
> > inconsistencies so they can probably be discarded, unless someone
> > disagrees.
> >
> > Configurable skips and limit maximums (E) may still be interesting.
> > Though, they don't necessarily have to be related to transactions, but
> > can instead be used to ensure streaming APIs are consumed in smaller
> > chunks.
> >
> > Cheers,
> > -Nick
> >
> >
> >
> > On Mon, Feb 24, 2020 at 7:26 AM Mike Rhodes <co...@dx13.co.uk>
wrote:
> >>
> >> Nick,
> >>
> >> Thanks for thinking this through, it's certainly subtle and very
unclear what is a "good" solution :(
> >>
> >> I have a couple of thoughts, firstly about the guarantees we
currently offer and then wondering whether there is an opportunity to
improve our API by offering a single guarantee across all request types
rather than bifurcating guarantees.
> >>
> >> ---
> >>
> >> The first point is that, by my reasoning, CouchDB 2.x doesn't
actually don't offer a point-in-time guarantee of the following sort
currently. I read this as your saying Couch does offer this guarantee,
apologies if I'm misreading:
> >>
> >>> Document the API behavior change that it may
> >>> present a view of the data is never a point-in-time[4] snapshot of
the
> >>> DB.
> >> ...
> >>> [4] For example they have a constraint that documents "a" and "z"
> >>> cannot both be in the database at the same time. But when iterating
> >>> it's possible that "a" was there at the start. Then by the end, "a"
> >>> was removed and "z" added, so both "a" and "z" would appear in the
> >>> emitted stream. Note that FoundationDB has APIs which exhibit the
same
> >>> "relaxed" constrains:
> >>>
https://urldefense.proofpoint.com/v2/url?u=https-3A__apple.github.io_foundationdb_api-2Dpython.html-23module-2Dfdb.locality&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=2FAZh1XhUm-BtJzY8Lca-ZC-76K6zLu2Q7zpFojCGk8&s=HzP2Pj3x5vl5LP656xQI81QM2YyZZuPN1HYbw_L7jzE&e=
> >>
> >> I don't believe we offer this guarantee because different database
shards will respond to the scatter-gather inherent to a single global
query type request at different times. This means that, given the
following sequence of events:
> >>
> >> (1) The shard containing "a" may start returning at time N.
> >> (2) "a" may be deleted at N+1, but (1) will still be streaming from
time N.
> >> (3) "z" may be written to a second shard at time N+2.
> >> (4) that second shard may not start returning until time N+3.
> >>
> >> By my reasoning, "a" and "z" could thus appear in the same result set
in current CouchDB, even if they never actually appear in the primary data
at the same time (regardless of latency of shard replicas coming into
agreement), voiding [4].
> >>
> >> By my reckoning, you have point-in-time across a query request when
you are working with a single shard, meaning we do have point in time for
two scenarios:
> >>
> >> - Partitioned queries.
> >> - Q=1 databases.
> >>
> >> Albeit this guarantee is still talking about the point in time of a
single shard's replica rather than all replicas, meaning that further
requests may produce different results if the shards are not in agreement.
Which can perhaps be fixed by using stable=true.
> >>
> >> I _think_ the working here is correct, but I'd welcome corrections in
my understanding!
> >>
> >> ---
> >>
> >> Our current behaviour seems extremely subtle and, I'd argue,
unexpected. It is hard to reason about if you really need a particular
guarantee.
> >>
> >> Is there an opportunity to clarify behaviour here, such that we
really _do_ guarantee point-in-time within _any_ single request, but only
do this by leveraging FoundationDB's transaction isolation semantics and
as such are only able to offer this based on the 5s timeout in place? The
request boundary offers a very clear cut, user-visible boundary. This
would obviously need to cover reads/writes of single docs and so on as
well as probably needing further work w.r.t. bulk docs etc.
> >>
> >> This restriction may naturally loosen as FoundationDB improves and
the 5s timeout may be increased.
> >>
> >> In this approach, my preference would be to add a closing line to the
result stream to contain both a bookmark (based on the FoundationDB key
perhaps rather than the index key of itself to avoid problems with
skip/limit?) and a complete/not-complete boolean to enable clients to
avoid the extra HTTP round-trip for completed result sets that Nick
mentions.
> >>
> >> ---
> >>
> >> For option (F), I feel that the "it sometimes works and sometimes
doesn't" effect of checking the update-seq to see if we can continue
streaming will be a confusing experience. I also find something similar
with option (A) where a single request covers potentially many points in
time and so feels hard to reason about, although it's a bit less subtle
than today.
> >>
> >> Footnote [2] seems quite a major problem, however, with the single
transaction approach and as Nick says, it is hard to pick "good" maximums
for skip -- perhaps users need to just avoid use of these in the new
system given its behaviour? It feels like there's a definite "against the
grain" aspect to these.
> >>
> >> --
> >> Mike.
> >>
> >> On Wed, 19 Feb 2020, at 22:39, Nick Vatamaniuc wrote:
> >>> Hello everyone,
> >>>
> >>> I'd like to discuss the shape and behavior of streaming APIs for
CouchDB 4.x
> >>>
> >>> By "streaming APIs" I mean APIs which stream data in row as it gets
> >>> read from the database. These are the endpoints I was thinking of:
> >>>
> >>> _all_docs, _all_dbs, _dbs_info and query results
> >>>
> >>> I want to focus on what happens when FoundationDB transactions
> >>> time-out after 5 seconds. Currently, all those APIs except
_changes[1]
> >>> feeds, will crash or freeze. The reason is because the
> >>> transaction_too_old error at the end of 5 seconds is retry-able by
> >>> default, so the request handlers run again and end up shoving the
> >>> whole request down the socket again, headers and all, which is
> >>> obviously broken and not what we want.
> >>>
> >>> There are few alternatives discussed in couchdb-dev channel. I'll
> >>> present some behaviors but feel free to add more. Some ideas might
> >>> have been discounted on the IRC discussion already but I'll present
> >>> them anyway in case is sparks further conversation:
> >>>
> >>> A) Do what _changes[1] feeds do. Start a new transaction and
continue
> >>> streaming the data from the next key after last emitted in the
> >>> previous transaction. Document the API behavior change that it may
> >>> present a view of the data is never a point-in-time[4] snapshot of
the
> >>> DB.
> >>>
> >>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> >>> don't have to change to continue using these CouchDB 4.0 endpoints
> >>> - This is the easiest to implement since it would re-use the
> >>> implementation for _changes feed (an extra option passed to the fold
> >>> function).
> >>> - Breaks API behavior if users relied on having a point-in-time[4]
> >>> snapshot view of the data.
> >>>
> >>> B) Simply end the stream. Let the users pass a `?transaction=true`
> >>> param which indicates they are aware the stream may end early and so
> >>> would have to paginate from the last emitted key with a skip=1. This
> >>> will keep the request bodies the same as current CouchDB. However,
if
> >>> the users got all the data one request, they will end up wasting
> >>> another request to see if there is more data available. If they
didn't
> >>> get any data they might have a too large of a skip value (see [2])
so
> >>> would have to guess different values for start/end keys. Or impose
max
> >>> limit for the `skip` parameter.
> >>>
> >>> C) End the stream and add a final metadata row like a "transaction":
> >>> "timeout" at the end. That will let the user know to keep paginating
> >>> from the last key onward. This won't work for `_all_dbs` and
> >>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> >>> feeds and only use this for views and and _all_docs? If we like this
> >>> choice, let's think what happens for those as I couldn't come up
with
> >>> anything decent there.
> >>>
> >>> D) Same as C but to solve the issue with skips[2], emit a bookmark
> >>> "key" of where the iteration stopped and the current "skip" and
> >>> "limit" params, which would keep decreasing. Then user would pass
> >>> those in "start_key=..." in the next request along with the limit
and
> >>> skip params. So something like "continuation":{"skip":599,
"limit":5,
> >>> "key":"..."}. This has the same issue with array results for
> >>> `_all_dbs` and `_dbs_info`[3].
> >>>
> >>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> >>> there such that response time is likely to fit in one transaction.
> >>> This could be tricky as different runtime environments will have
> >>> different characteristics. Also, if the timeout happens there isn't
a
> >>> a nice way to send an HTTP error since we already sent the 200
> >>> response. The downside is that this might break how some users use
the
> >>> API, if say the are using large skips and limits already. Perhaps
here
> >>> we do both B and D, such that if users want transactional behavior,
> >>> they specify that `transaction=true` param and only then we enforce
> >>> low limit and skip maximums.
> >>>
> >>> F) At least for `_all_docs` it seems providing a point-in-time
> >>> snapshot view doesn't necessarily need to be tied to transaction
> >>> boundaries. We could check the update sequence of the database at
the
> >>> start of the next transaction and if it hasn't changed we can
continue
> >>> emitting a consistent view. This can apply to C and D and would just
> >>> determine when the stream ends. If there are no writes happening to
> >>> the db, this could potential streams all the data just like option A
> >>> would do. Not entirely sure if this would work for views.
> >>>
> >>> So what do we think? I can see different combinations of options
here,
> >>> maybe even different for each API point. For example `_all_dbs`,
> >>> `_dbs_info` are always A, and `_all_docs` and views default to A but
> >>> have parameters to do F, etc.
> >>>
> >>> Cheers,
> >>> -Nick
> >>>
> >>> Some footnotes:
> >>>
> >>> [1] _changes feeds is the only one that works currently. It behaves
as
> >>> per RFC
> >>>
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb-2Ddocumentation_blob_master_rfcs_003-2Dfdb-2Dseq-2Dindex.md-23access-2Dpatterns&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=2FAZh1XhUm-BtJzY8Lca-ZC-76K6zLu2Q7zpFojCGk8&s=7Khj1Mm0BvTQkebpPs4O7kE2moJMmmGjV7_icfRo_q8&e=
.
> >>> That is, we continue streaming the data by resetting the transaction
> >>> object and restarting from the last emitted key (db sequence in this
> >>> case). However, because the transaction restarts if a document is
> >>> updated while the streaming take place, it may appear in the
_changes
> >>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> >>> have to document it, since previously we presented this
point-in-time
> >>> snapshot of the database from when we started streaming.
> >>>
> >>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> >>> currently support efficient offsets for key selectors
> >>> (
https://urldefense.proofpoint.com/v2/url?u=https-3A__apple.github.io_foundationdb_known-2Dlimitations.html-23dont-2Duse-2Dkey-2Dselectors-2Dfor-2Dpaging&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=2FAZh1XhUm-BtJzY8Lca-ZC-76K6zLu2Q7zpFojCGk8&s=iJJQDYHRGZ6PQ_FF3sy9nBSygRYkEh3cRPMFYg2Tkq8&e=
)
> >>> we implemented skip by iterating over the data. This means that a
skip
> >>> of say 100000 could keep timing out the transaction without yielding
> >>> any data.
> >>>
> >>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> >>> obvious place to insert a last metadata row.
> >>>
> >>> [4] For example they have a constraint that documents "a" and "z"
> >>> cannot both be in the database at the same time. But when iterating
> >>> it's possible that "a" was there at the start. Then by the end, "a"
> >>> was removed and "z" added, so both "a" and "z" would appear in the
> >>> emitted stream. Note that FoundationDB has APIs which exhibit the
same
> >>> "relaxed" constrains:
> >>>
https://urldefense.proofpoint.com/v2/url?u=https-3A__apple.github.io_foundationdb_api-2Dpython.html-23module-2Dfdb.locality&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=2FAZh1XhUm-BtJzY8Lca-ZC-76K6zLu2Q7zpFojCGk8&s=HzP2Pj3x5vl5LP656xQI81QM2YyZZuPN1HYbw_L7jzE&e=
> >>>
>
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Nick Vatamaniuc <va...@gmail.com>.
Thanks for replying, Adam!
Thinking about it some more, it seems there are two benefits
to changing the streaming APIs:
1) To provide users with a serializable snapshot. We don't currently
have that, as Mike pointed out, unless we use n=1&q=1 or CouchDB
version 1.x. It would be nice to get that with a new release.
2) To avoid the general anti-pattern of streaming all the data in one
single request or using a very large skip or limit values.
However, I think the two improvements are not necessarily tied to each
other. For example, we could set configurable mandatory max limits
(option E) for all the streaming endpoints in 3.x as well. On the
other hand, even with a single transaction we could stream say 150k
rows in 5 seconds. If at some future point FDB would allow minute long
transactions, we could stream millions of rows before timing out and
it would still not be a desirable pattern of usage. This is also
basically option F in a read-only case (we can emit a snapshot as long
as there are no writes to that particular db), and I think we agree
that it is not that appealing of an option.
What do we think then about having per-endpoint configurable max
limits (option E). The configuration could look something like:
[request_limits]
all_docs = 5000
views = 2500
list_dbs = 1000
dbs_info = 500
If those limits are set, and a request is made against an endpoint
without the limit parameter, or a limit or skip is provided but is
greater than the maximum, it would return back immediately with an
error (412) and an indication of what the a max limit value is.
And I agree that client libraries are important in helping here.
So for example, Cloudant client libraries could detect that error and
either return it to the user, or, as a compatibility mode, use a
few consecutive requests behind the scenes to stream all the data back
to the user as requested without the users application code needing
any updates at all.
If those limits are not set, the API would behave as it does now. This
would provide a smoother upgrade path from 3.x to 4.x so users
wouldn't have to rewrite their applications.
I still think the bookmarking approach is interesting, but I think it
might work better for a new API endpoint or enabled with an explicit
parameter. I can see a case where users might be using Python requests
library to fetch _all_docs, assuming it gets all the data, then after
the upgrade same API endpoint suddenly returns only a fraction of the
rows. There might be another "bookmark" field there but it is buried
under a few application layers and gets ignored. They users just notice
the missing data at some point, and it could be perceived as data loss
in a sense.
> how much value do we derive from that streaming behavior if we
aggressively limit the `limit`?
Oh good point! That makes sense. We might be able to simplify
quite a bit of logic internally if we didn't actually stream the data.
We buffer thousands of doc updates for _bulk_docs already so perhaps
it is not that different doing it when reading data in these APIs as well.
It is something that we'd have to experiment with and see how it would
behaves.
-Nick
On Wed, Apr 1, 2020 at 9:07 PM Adam Kocoloski <ko...@apache.org> wrote:
>
> This is a really important topic; thanks Nick for bringing it up. Sorry I didn’t comment earlier. I think Mike neatly captures my perspective with this bit:
>
> >> Our current behaviour seems extremely subtle and, I'd argue, unexpected. It is hard to reason about if you really need a particular guarantee.
> >>
> >> Is there an opportunity to clarify behaviour here, such that we really _do_ guarantee point-in-time within _any_ single request, but only do this by leveraging FoundationDB's transaction isolation semantics and as such are only able to offer this based on the 5s timeout in place? The request boundary offers a very clear cut, user-visible boundary. This would obviously need to cover reads/writes of single docs and so on as well as probably needing further work w.r.t. bulk docs etc.
> >>
> >> This restriction may naturally loosen as FoundationDB improves and the 5s timeout may be increased.
>
> It’d be great if we could agree on this use of serializable snapshot isolation under the hood for each response to a CouchDB API request (excepting _changes) as an optimal state.
>
> Of course, we have this complicating factor of an existing API and a community of users running applications in production against that API :) As you can imagine from the above, I’d be opposed to A); I think that squanders a real opportunity that we have here with a new major version. I also think that the return on investment for F) is too low; a large portion of our production databases see a 24/7 write load so a code path that only activates when a DB is quiesced doesn’t get my vote.
>
> When I look at the other options, I think it’s important to take a broader view and consider the user experience in the client libraries as well as the API. Our experience at IBM Cloud is that a large majority of API requests come from a well-defined set of client libraries, and as we consider non-trivial changes to the API we can look to those libraries as a way to smooth over the API breakage, and intelligently surface new capabilities even if the least-disruptive way to introduce them to the API is a bit janky.
>
> As a concrete example, I would support an aggressive ceiling on `limit` and `skip` in the 4.0 API, while enhancing popular client libraries as needed to allow users to opt-in to automatic pagination through larger result sets.
>
> Nick rightly points out that we don’t have a good way to declare a read version timeout when we’ve already streamed a portion of the result set to the client, which is something we ought consider even if we do apply the restrictions in E). I acknowledge that I may be opening a can of worms, but ... how much value do we derive from that streaming behavior if we aggressively limit the `limit`? We wouldn’t be holding that much data in memory on the CouchDB side, and I don’t think many of our clients are parsing half-completed JSON objects for anything beyond the _changes feed. Something to think about.
>
> Cheers, Adam
>
> > On Feb 25, 2020, at 2:52 PM, Nick Vatamaniuc <va...@gmail.com> wrote:
> >
> > Hi Mike,
> >
> > Good point about CouchDB not actually providing point-in-time
> > snapshots. I missed those cases when thinking about it.
> >
> > I wonder if that points to defaulting to option A since it maintains
> > the API compatibility and doesn't loosen the current constraints
> > anyway. At least it will un-break the current version of the branch
> > until we figure out something better. Otherwise it's completely
> > unusable for dbs with more than 200-300k documents.
> >
> > I like the idea of returning a bookmark and a completed/not-completed
> > flag. That is, it would be option D for _all_docs and map-reduce
> > views, but instead of the complex continuation object it would be a
> > base64-encoded, opaque object. Passing a bookmark back in as a
> > parameter would be exclusive to passing in a start, end, skip, limit,
> > and direction (forward/reverse) parameters. For _all_dbs, and
> > _dbs_info where we don't have a place for metadata rows, we might need
> > a new API endpoint. And maybe that opens the door to expose more
> > transactional features in the API in general...
> >
> > Also, it seems B, C and F have too many corner cases and
> > inconsistencies so they can probably be discarded, unless someone
> > disagrees.
> >
> > Configurable skips and limit maximums (E) may still be interesting.
> > Though, they don't necessarily have to be related to transactions, but
> > can instead be used to ensure streaming APIs are consumed in smaller
> > chunks.
> >
> > Cheers,
> > -Nick
> >
> >
> >
> > On Mon, Feb 24, 2020 at 7:26 AM Mike Rhodes <co...@dx13.co.uk> wrote:
> >>
> >> Nick,
> >>
> >> Thanks for thinking this through, it's certainly subtle and very unclear what is a "good" solution :(
> >>
> >> I have a couple of thoughts, firstly about the guarantees we currently offer and then wondering whether there is an opportunity to improve our API by offering a single guarantee across all request types rather than bifurcating guarantees.
> >>
> >> ---
> >>
> >> The first point is that, by my reasoning, CouchDB 2.x doesn't actually don't offer a point-in-time guarantee of the following sort currently. I read this as your saying Couch does offer this guarantee, apologies if I'm misreading:
> >>
> >>> Document the API behavior change that it may
> >>> present a view of the data is never a point-in-time[4] snapshot of the
> >>> DB.
> >> ...
> >>> [4] For example they have a constraint that documents "a" and "z"
> >>> cannot both be in the database at the same time. But when iterating
> >>> it's possible that "a" was there at the start. Then by the end, "a"
> >>> was removed and "z" added, so both "a" and "z" would appear in the
> >>> emitted stream. Note that FoundationDB has APIs which exhibit the same
> >>> "relaxed" constrains:
> >>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >>
> >> I don't believe we offer this guarantee because different database shards will respond to the scatter-gather inherent to a single global query type request at different times. This means that, given the following sequence of events:
> >>
> >> (1) The shard containing "a" may start returning at time N.
> >> (2) "a" may be deleted at N+1, but (1) will still be streaming from time N.
> >> (3) "z" may be written to a second shard at time N+2.
> >> (4) that second shard may not start returning until time N+3.
> >>
> >> By my reasoning, "a" and "z" could thus appear in the same result set in current CouchDB, even if they never actually appear in the primary data at the same time (regardless of latency of shard replicas coming into agreement), voiding [4].
> >>
> >> By my reckoning, you have point-in-time across a query request when you are working with a single shard, meaning we do have point in time for two scenarios:
> >>
> >> - Partitioned queries.
> >> - Q=1 databases.
> >>
> >> Albeit this guarantee is still talking about the point in time of a single shard's replica rather than all replicas, meaning that further requests may produce different results if the shards are not in agreement. Which can perhaps be fixed by using stable=true.
> >>
> >> I _think_ the working here is correct, but I'd welcome corrections in my understanding!
> >>
> >> ---
> >>
> >> Our current behaviour seems extremely subtle and, I'd argue, unexpected. It is hard to reason about if you really need a particular guarantee.
> >>
> >> Is there an opportunity to clarify behaviour here, such that we really _do_ guarantee point-in-time within _any_ single request, but only do this by leveraging FoundationDB's transaction isolation semantics and as such are only able to offer this based on the 5s timeout in place? The request boundary offers a very clear cut, user-visible boundary. This would obviously need to cover reads/writes of single docs and so on as well as probably needing further work w.r.t. bulk docs etc.
> >>
> >> This restriction may naturally loosen as FoundationDB improves and the 5s timeout may be increased.
> >>
> >> In this approach, my preference would be to add a closing line to the result stream to contain both a bookmark (based on the FoundationDB key perhaps rather than the index key of itself to avoid problems with skip/limit?) and a complete/not-complete boolean to enable clients to avoid the extra HTTP round-trip for completed result sets that Nick mentions.
> >>
> >> ---
> >>
> >> For option (F), I feel that the "it sometimes works and sometimes doesn't" effect of checking the update-seq to see if we can continue streaming will be a confusing experience. I also find something similar with option (A) where a single request covers potentially many points in time and so feels hard to reason about, although it's a bit less subtle than today.
> >>
> >> Footnote [2] seems quite a major problem, however, with the single transaction approach and as Nick says, it is hard to pick "good" maximums for skip -- perhaps users need to just avoid use of these in the new system given its behaviour? It feels like there's a definite "against the grain" aspect to these.
> >>
> >> --
> >> Mike.
> >>
> >> On Wed, 19 Feb 2020, at 22:39, Nick Vatamaniuc wrote:
> >>> Hello everyone,
> >>>
> >>> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> >>>
> >>> By "streaming APIs" I mean APIs which stream data in row as it gets
> >>> read from the database. These are the endpoints I was thinking of:
> >>>
> >>> _all_docs, _all_dbs, _dbs_info and query results
> >>>
> >>> I want to focus on what happens when FoundationDB transactions
> >>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> >>> feeds, will crash or freeze. The reason is because the
> >>> transaction_too_old error at the end of 5 seconds is retry-able by
> >>> default, so the request handlers run again and end up shoving the
> >>> whole request down the socket again, headers and all, which is
> >>> obviously broken and not what we want.
> >>>
> >>> There are few alternatives discussed in couchdb-dev channel. I'll
> >>> present some behaviors but feel free to add more. Some ideas might
> >>> have been discounted on the IRC discussion already but I'll present
> >>> them anyway in case is sparks further conversation:
> >>>
> >>> A) Do what _changes[1] feeds do. Start a new transaction and continue
> >>> streaming the data from the next key after last emitted in the
> >>> previous transaction. Document the API behavior change that it may
> >>> present a view of the data is never a point-in-time[4] snapshot of the
> >>> DB.
> >>>
> >>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> >>> don't have to change to continue using these CouchDB 4.0 endpoints
> >>> - This is the easiest to implement since it would re-use the
> >>> implementation for _changes feed (an extra option passed to the fold
> >>> function).
> >>> - Breaks API behavior if users relied on having a point-in-time[4]
> >>> snapshot view of the data.
> >>>
> >>> B) Simply end the stream. Let the users pass a `?transaction=true`
> >>> param which indicates they are aware the stream may end early and so
> >>> would have to paginate from the last emitted key with a skip=1. This
> >>> will keep the request bodies the same as current CouchDB. However, if
> >>> the users got all the data one request, they will end up wasting
> >>> another request to see if there is more data available. If they didn't
> >>> get any data they might have a too large of a skip value (see [2]) so
> >>> would have to guess different values for start/end keys. Or impose max
> >>> limit for the `skip` parameter.
> >>>
> >>> C) End the stream and add a final metadata row like a "transaction":
> >>> "timeout" at the end. That will let the user know to keep paginating
> >>> from the last key onward. This won't work for `_all_dbs` and
> >>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> >>> feeds and only use this for views and and _all_docs? If we like this
> >>> choice, let's think what happens for those as I couldn't come up with
> >>> anything decent there.
> >>>
> >>> D) Same as C but to solve the issue with skips[2], emit a bookmark
> >>> "key" of where the iteration stopped and the current "skip" and
> >>> "limit" params, which would keep decreasing. Then user would pass
> >>> those in "start_key=..." in the next request along with the limit and
> >>> skip params. So something like "continuation":{"skip":599, "limit":5,
> >>> "key":"..."}. This has the same issue with array results for
> >>> `_all_dbs` and `_dbs_info`[3].
> >>>
> >>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> >>> there such that response time is likely to fit in one transaction.
> >>> This could be tricky as different runtime environments will have
> >>> different characteristics. Also, if the timeout happens there isn't a
> >>> a nice way to send an HTTP error since we already sent the 200
> >>> response. The downside is that this might break how some users use the
> >>> API, if say the are using large skips and limits already. Perhaps here
> >>> we do both B and D, such that if users want transactional behavior,
> >>> they specify that `transaction=true` param and only then we enforce
> >>> low limit and skip maximums.
> >>>
> >>> F) At least for `_all_docs` it seems providing a point-in-time
> >>> snapshot view doesn't necessarily need to be tied to transaction
> >>> boundaries. We could check the update sequence of the database at the
> >>> start of the next transaction and if it hasn't changed we can continue
> >>> emitting a consistent view. This can apply to C and D and would just
> >>> determine when the stream ends. If there are no writes happening to
> >>> the db, this could potential streams all the data just like option A
> >>> would do. Not entirely sure if this would work for views.
> >>>
> >>> So what do we think? I can see different combinations of options here,
> >>> maybe even different for each API point. For example `_all_dbs`,
> >>> `_dbs_info` are always A, and `_all_docs` and views default to A but
> >>> have parameters to do F, etc.
> >>>
> >>> Cheers,
> >>> -Nick
> >>>
> >>> Some footnotes:
> >>>
> >>> [1] _changes feeds is the only one that works currently. It behaves as
> >>> per RFC
> >>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> >>> That is, we continue streaming the data by resetting the transaction
> >>> object and restarting from the last emitted key (db sequence in this
> >>> case). However, because the transaction restarts if a document is
> >>> updated while the streaming take place, it may appear in the _changes
> >>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> >>> have to document it, since previously we presented this point-in-time
> >>> snapshot of the database from when we started streaming.
> >>>
> >>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> >>> currently support efficient offsets for key selectors
> >>> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> >>> we implemented skip by iterating over the data. This means that a skip
> >>> of say 100000 could keep timing out the transaction without yielding
> >>> any data.
> >>>
> >>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> >>> obvious place to insert a last metadata row.
> >>>
> >>> [4] For example they have a constraint that documents "a" and "z"
> >>> cannot both be in the database at the same time. But when iterating
> >>> it's possible that "a" was there at the start. Then by the end, "a"
> >>> was removed and "z" added, so both "a" and "z" would appear in the
> >>> emitted stream. Note that FoundationDB has APIs which exhibit the same
> >>> "relaxed" constrains:
> >>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >>>
>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Adam Kocoloski <ko...@apache.org>.
This is a really important topic; thanks Nick for bringing it up. Sorry I didn’t comment earlier. I think Mike neatly captures my perspective with this bit:
>> Our current behaviour seems extremely subtle and, I'd argue, unexpected. It is hard to reason about if you really need a particular guarantee.
>>
>> Is there an opportunity to clarify behaviour here, such that we really _do_ guarantee point-in-time within _any_ single request, but only do this by leveraging FoundationDB's transaction isolation semantics and as such are only able to offer this based on the 5s timeout in place? The request boundary offers a very clear cut, user-visible boundary. This would obviously need to cover reads/writes of single docs and so on as well as probably needing further work w.r.t. bulk docs etc.
>>
>> This restriction may naturally loosen as FoundationDB improves and the 5s timeout may be increased.
It’d be great if we could agree on this use of serializable snapshot isolation under the hood for each response to a CouchDB API request (excepting _changes) as an optimal state.
Of course, we have this complicating factor of an existing API and a community of users running applications in production against that API :) As you can imagine from the above, I’d be opposed to A); I think that squanders a real opportunity that we have here with a new major version. I also think that the return on investment for F) is too low; a large portion of our production databases see a 24/7 write load so a code path that only activates when a DB is quiesced doesn’t get my vote.
When I look at the other options, I think it’s important to take a broader view and consider the user experience in the client libraries as well as the API. Our experience at IBM Cloud is that a large majority of API requests come from a well-defined set of client libraries, and as we consider non-trivial changes to the API we can look to those libraries as a way to smooth over the API breakage, and intelligently surface new capabilities even if the least-disruptive way to introduce them to the API is a bit janky.
As a concrete example, I would support an aggressive ceiling on `limit` and `skip` in the 4.0 API, while enhancing popular client libraries as needed to allow users to opt-in to automatic pagination through larger result sets.
Nick rightly points out that we don’t have a good way to declare a read version timeout when we’ve already streamed a portion of the result set to the client, which is something we ought consider even if we do apply the restrictions in E). I acknowledge that I may be opening a can of worms, but ... how much value do we derive from that streaming behavior if we aggressively limit the `limit`? We wouldn’t be holding that much data in memory on the CouchDB side, and I don’t think many of our clients are parsing half-completed JSON objects for anything beyond the _changes feed. Something to think about.
Cheers, Adam
> On Feb 25, 2020, at 2:52 PM, Nick Vatamaniuc <va...@gmail.com> wrote:
>
> Hi Mike,
>
> Good point about CouchDB not actually providing point-in-time
> snapshots. I missed those cases when thinking about it.
>
> I wonder if that points to defaulting to option A since it maintains
> the API compatibility and doesn't loosen the current constraints
> anyway. At least it will un-break the current version of the branch
> until we figure out something better. Otherwise it's completely
> unusable for dbs with more than 200-300k documents.
>
> I like the idea of returning a bookmark and a completed/not-completed
> flag. That is, it would be option D for _all_docs and map-reduce
> views, but instead of the complex continuation object it would be a
> base64-encoded, opaque object. Passing a bookmark back in as a
> parameter would be exclusive to passing in a start, end, skip, limit,
> and direction (forward/reverse) parameters. For _all_dbs, and
> _dbs_info where we don't have a place for metadata rows, we might need
> a new API endpoint. And maybe that opens the door to expose more
> transactional features in the API in general...
>
> Also, it seems B, C and F have too many corner cases and
> inconsistencies so they can probably be discarded, unless someone
> disagrees.
>
> Configurable skips and limit maximums (E) may still be interesting.
> Though, they don't necessarily have to be related to transactions, but
> can instead be used to ensure streaming APIs are consumed in smaller
> chunks.
>
> Cheers,
> -Nick
>
>
>
> On Mon, Feb 24, 2020 at 7:26 AM Mike Rhodes <co...@dx13.co.uk> wrote:
>>
>> Nick,
>>
>> Thanks for thinking this through, it's certainly subtle and very unclear what is a "good" solution :(
>>
>> I have a couple of thoughts, firstly about the guarantees we currently offer and then wondering whether there is an opportunity to improve our API by offering a single guarantee across all request types rather than bifurcating guarantees.
>>
>> ---
>>
>> The first point is that, by my reasoning, CouchDB 2.x doesn't actually don't offer a point-in-time guarantee of the following sort currently. I read this as your saying Couch does offer this guarantee, apologies if I'm misreading:
>>
>>> Document the API behavior change that it may
>>> present a view of the data is never a point-in-time[4] snapshot of the
>>> DB.
>> ...
>>> [4] For example they have a constraint that documents "a" and "z"
>>> cannot both be in the database at the same time. But when iterating
>>> it's possible that "a" was there at the start. Then by the end, "a"
>>> was removed and "z" added, so both "a" and "z" would appear in the
>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>> "relaxed" constrains:
>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>
>> I don't believe we offer this guarantee because different database shards will respond to the scatter-gather inherent to a single global query type request at different times. This means that, given the following sequence of events:
>>
>> (1) The shard containing "a" may start returning at time N.
>> (2) "a" may be deleted at N+1, but (1) will still be streaming from time N.
>> (3) "z" may be written to a second shard at time N+2.
>> (4) that second shard may not start returning until time N+3.
>>
>> By my reasoning, "a" and "z" could thus appear in the same result set in current CouchDB, even if they never actually appear in the primary data at the same time (regardless of latency of shard replicas coming into agreement), voiding [4].
>>
>> By my reckoning, you have point-in-time across a query request when you are working with a single shard, meaning we do have point in time for two scenarios:
>>
>> - Partitioned queries.
>> - Q=1 databases.
>>
>> Albeit this guarantee is still talking about the point in time of a single shard's replica rather than all replicas, meaning that further requests may produce different results if the shards are not in agreement. Which can perhaps be fixed by using stable=true.
>>
>> I _think_ the working here is correct, but I'd welcome corrections in my understanding!
>>
>> ---
>>
>> Our current behaviour seems extremely subtle and, I'd argue, unexpected. It is hard to reason about if you really need a particular guarantee.
>>
>> Is there an opportunity to clarify behaviour here, such that we really _do_ guarantee point-in-time within _any_ single request, but only do this by leveraging FoundationDB's transaction isolation semantics and as such are only able to offer this based on the 5s timeout in place? The request boundary offers a very clear cut, user-visible boundary. This would obviously need to cover reads/writes of single docs and so on as well as probably needing further work w.r.t. bulk docs etc.
>>
>> This restriction may naturally loosen as FoundationDB improves and the 5s timeout may be increased.
>>
>> In this approach, my preference would be to add a closing line to the result stream to contain both a bookmark (based on the FoundationDB key perhaps rather than the index key of itself to avoid problems with skip/limit?) and a complete/not-complete boolean to enable clients to avoid the extra HTTP round-trip for completed result sets that Nick mentions.
>>
>> ---
>>
>> For option (F), I feel that the "it sometimes works and sometimes doesn't" effect of checking the update-seq to see if we can continue streaming will be a confusing experience. I also find something similar with option (A) where a single request covers potentially many points in time and so feels hard to reason about, although it's a bit less subtle than today.
>>
>> Footnote [2] seems quite a major problem, however, with the single transaction approach and as Nick says, it is hard to pick "good" maximums for skip -- perhaps users need to just avoid use of these in the new system given its behaviour? It feels like there's a definite "against the grain" aspect to these.
>>
>> --
>> Mike.
>>
>> On Wed, 19 Feb 2020, at 22:39, Nick Vatamaniuc wrote:
>>> Hello everyone,
>>>
>>> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>>>
>>> By "streaming APIs" I mean APIs which stream data in row as it gets
>>> read from the database. These are the endpoints I was thinking of:
>>>
>>> _all_docs, _all_dbs, _dbs_info and query results
>>>
>>> I want to focus on what happens when FoundationDB transactions
>>> time-out after 5 seconds. Currently, all those APIs except _changes[1]
>>> feeds, will crash or freeze. The reason is because the
>>> transaction_too_old error at the end of 5 seconds is retry-able by
>>> default, so the request handlers run again and end up shoving the
>>> whole request down the socket again, headers and all, which is
>>> obviously broken and not what we want.
>>>
>>> There are few alternatives discussed in couchdb-dev channel. I'll
>>> present some behaviors but feel free to add more. Some ideas might
>>> have been discounted on the IRC discussion already but I'll present
>>> them anyway in case is sparks further conversation:
>>>
>>> A) Do what _changes[1] feeds do. Start a new transaction and continue
>>> streaming the data from the next key after last emitted in the
>>> previous transaction. Document the API behavior change that it may
>>> present a view of the data is never a point-in-time[4] snapshot of the
>>> DB.
>>>
>>> - Keeps the API shape the same as CouchDB <4.0. Client libraries
>>> don't have to change to continue using these CouchDB 4.0 endpoints
>>> - This is the easiest to implement since it would re-use the
>>> implementation for _changes feed (an extra option passed to the fold
>>> function).
>>> - Breaks API behavior if users relied on having a point-in-time[4]
>>> snapshot view of the data.
>>>
>>> B) Simply end the stream. Let the users pass a `?transaction=true`
>>> param which indicates they are aware the stream may end early and so
>>> would have to paginate from the last emitted key with a skip=1. This
>>> will keep the request bodies the same as current CouchDB. However, if
>>> the users got all the data one request, they will end up wasting
>>> another request to see if there is more data available. If they didn't
>>> get any data they might have a too large of a skip value (see [2]) so
>>> would have to guess different values for start/end keys. Or impose max
>>> limit for the `skip` parameter.
>>>
>>> C) End the stream and add a final metadata row like a "transaction":
>>> "timeout" at the end. That will let the user know to keep paginating
>>> from the last key onward. This won't work for `_all_dbs` and
>>> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
>>> feeds and only use this for views and and _all_docs? If we like this
>>> choice, let's think what happens for those as I couldn't come up with
>>> anything decent there.
>>>
>>> D) Same as C but to solve the issue with skips[2], emit a bookmark
>>> "key" of where the iteration stopped and the current "skip" and
>>> "limit" params, which would keep decreasing. Then user would pass
>>> those in "start_key=..." in the next request along with the limit and
>>> skip params. So something like "continuation":{"skip":599, "limit":5,
>>> "key":"..."}. This has the same issue with array results for
>>> `_all_dbs` and `_dbs_info`[3].
>>>
>>> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
>>> there such that response time is likely to fit in one transaction.
>>> This could be tricky as different runtime environments will have
>>> different characteristics. Also, if the timeout happens there isn't a
>>> a nice way to send an HTTP error since we already sent the 200
>>> response. The downside is that this might break how some users use the
>>> API, if say the are using large skips and limits already. Perhaps here
>>> we do both B and D, such that if users want transactional behavior,
>>> they specify that `transaction=true` param and only then we enforce
>>> low limit and skip maximums.
>>>
>>> F) At least for `_all_docs` it seems providing a point-in-time
>>> snapshot view doesn't necessarily need to be tied to transaction
>>> boundaries. We could check the update sequence of the database at the
>>> start of the next transaction and if it hasn't changed we can continue
>>> emitting a consistent view. This can apply to C and D and would just
>>> determine when the stream ends. If there are no writes happening to
>>> the db, this could potential streams all the data just like option A
>>> would do. Not entirely sure if this would work for views.
>>>
>>> So what do we think? I can see different combinations of options here,
>>> maybe even different for each API point. For example `_all_dbs`,
>>> `_dbs_info` are always A, and `_all_docs` and views default to A but
>>> have parameters to do F, etc.
>>>
>>> Cheers,
>>> -Nick
>>>
>>> Some footnotes:
>>>
>>> [1] _changes feeds is the only one that works currently. It behaves as
>>> per RFC
>>> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
>>> That is, we continue streaming the data by resetting the transaction
>>> object and restarting from the last emitted key (db sequence in this
>>> case). However, because the transaction restarts if a document is
>>> updated while the streaming take place, it may appear in the _changes
>>> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
>>> have to document it, since previously we presented this point-in-time
>>> snapshot of the database from when we started streaming.
>>>
>>> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
>>> currently support efficient offsets for key selectors
>>> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
>>> we implemented skip by iterating over the data. This means that a skip
>>> of say 100000 could keep timing out the transaction without yielding
>>> any data.
>>>
>>> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
>>> obvious place to insert a last metadata row.
>>>
>>> [4] For example they have a constraint that documents "a" and "z"
>>> cannot both be in the database at the same time. But when iterating
>>> it's possible that "a" was there at the start. Then by the end, "a"
>>> was removed and "z" added, so both "a" and "z" would appear in the
>>> emitted stream. Note that FoundationDB has APIs which exhibit the same
>>> "relaxed" constrains:
>>> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>>>
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Nick Vatamaniuc <va...@gmail.com>.
Hi Mike,
Good point about CouchDB not actually providing point-in-time
snapshots. I missed those cases when thinking about it.
I wonder if that points to defaulting to option A since it maintains
the API compatibility and doesn't loosen the current constraints
anyway. At least it will un-break the current version of the branch
until we figure out something better. Otherwise it's completely
unusable for dbs with more than 200-300k documents.
I like the idea of returning a bookmark and a completed/not-completed
flag. That is, it would be option D for _all_docs and map-reduce
views, but instead of the complex continuation object it would be a
base64-encoded, opaque object. Passing a bookmark back in as a
parameter would be exclusive to passing in a start, end, skip, limit,
and direction (forward/reverse) parameters. For _all_dbs, and
_dbs_info where we don't have a place for metadata rows, we might need
a new API endpoint. And maybe that opens the door to expose more
transactional features in the API in general...
Also, it seems B, C and F have too many corner cases and
inconsistencies so they can probably be discarded, unless someone
disagrees.
Configurable skips and limit maximums (E) may still be interesting.
Though, they don't necessarily have to be related to transactions, but
can instead be used to ensure streaming APIs are consumed in smaller
chunks.
Cheers,
-Nick
On Mon, Feb 24, 2020 at 7:26 AM Mike Rhodes <co...@dx13.co.uk> wrote:
>
> Nick,
>
> Thanks for thinking this through, it's certainly subtle and very unclear what is a "good" solution :(
>
> I have a couple of thoughts, firstly about the guarantees we currently offer and then wondering whether there is an opportunity to improve our API by offering a single guarantee across all request types rather than bifurcating guarantees.
>
> ---
>
> The first point is that, by my reasoning, CouchDB 2.x doesn't actually don't offer a point-in-time guarantee of the following sort currently. I read this as your saying Couch does offer this guarantee, apologies if I'm misreading:
>
> > Document the API behavior change that it may
> > present a view of the data is never a point-in-time[4] snapshot of the
> > DB.
> ...
> > [4] For example they have a constraint that documents "a" and "z"
> > cannot both be in the database at the same time. But when iterating
> > it's possible that "a" was there at the start. Then by the end, "a"
> > was removed and "z" added, so both "a" and "z" would appear in the
> > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > "relaxed" constrains:
> > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>
> I don't believe we offer this guarantee because different database shards will respond to the scatter-gather inherent to a single global query type request at different times. This means that, given the following sequence of events:
>
> (1) The shard containing "a" may start returning at time N.
> (2) "a" may be deleted at N+1, but (1) will still be streaming from time N.
> (3) "z" may be written to a second shard at time N+2.
> (4) that second shard may not start returning until time N+3.
>
> By my reasoning, "a" and "z" could thus appear in the same result set in current CouchDB, even if they never actually appear in the primary data at the same time (regardless of latency of shard replicas coming into agreement), voiding [4].
>
> By my reckoning, you have point-in-time across a query request when you are working with a single shard, meaning we do have point in time for two scenarios:
>
> - Partitioned queries.
> - Q=1 databases.
>
> Albeit this guarantee is still talking about the point in time of a single shard's replica rather than all replicas, meaning that further requests may produce different results if the shards are not in agreement. Which can perhaps be fixed by using stable=true.
>
> I _think_ the working here is correct, but I'd welcome corrections in my understanding!
>
> ---
>
> Our current behaviour seems extremely subtle and, I'd argue, unexpected. It is hard to reason about if you really need a particular guarantee.
>
> Is there an opportunity to clarify behaviour here, such that we really _do_ guarantee point-in-time within _any_ single request, but only do this by leveraging FoundationDB's transaction isolation semantics and as such are only able to offer this based on the 5s timeout in place? The request boundary offers a very clear cut, user-visible boundary. This would obviously need to cover reads/writes of single docs and so on as well as probably needing further work w.r.t. bulk docs etc.
>
> This restriction may naturally loosen as FoundationDB improves and the 5s timeout may be increased.
>
> In this approach, my preference would be to add a closing line to the result stream to contain both a bookmark (based on the FoundationDB key perhaps rather than the index key of itself to avoid problems with skip/limit?) and a complete/not-complete boolean to enable clients to avoid the extra HTTP round-trip for completed result sets that Nick mentions.
>
> ---
>
> For option (F), I feel that the "it sometimes works and sometimes doesn't" effect of checking the update-seq to see if we can continue streaming will be a confusing experience. I also find something similar with option (A) where a single request covers potentially many points in time and so feels hard to reason about, although it's a bit less subtle than today.
>
> Footnote [2] seems quite a major problem, however, with the single transaction approach and as Nick says, it is hard to pick "good" maximums for skip -- perhaps users need to just avoid use of these in the new system given its behaviour? It feels like there's a definite "against the grain" aspect to these.
>
> --
> Mike.
>
> On Wed, 19 Feb 2020, at 22:39, Nick Vatamaniuc wrote:
> > Hello everyone,
> >
> > I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
> >
> > By "streaming APIs" I mean APIs which stream data in row as it gets
> > read from the database. These are the endpoints I was thinking of:
> >
> > _all_docs, _all_dbs, _dbs_info and query results
> >
> > I want to focus on what happens when FoundationDB transactions
> > time-out after 5 seconds. Currently, all those APIs except _changes[1]
> > feeds, will crash or freeze. The reason is because the
> > transaction_too_old error at the end of 5 seconds is retry-able by
> > default, so the request handlers run again and end up shoving the
> > whole request down the socket again, headers and all, which is
> > obviously broken and not what we want.
> >
> > There are few alternatives discussed in couchdb-dev channel. I'll
> > present some behaviors but feel free to add more. Some ideas might
> > have been discounted on the IRC discussion already but I'll present
> > them anyway in case is sparks further conversation:
> >
> > A) Do what _changes[1] feeds do. Start a new transaction and continue
> > streaming the data from the next key after last emitted in the
> > previous transaction. Document the API behavior change that it may
> > present a view of the data is never a point-in-time[4] snapshot of the
> > DB.
> >
> > - Keeps the API shape the same as CouchDB <4.0. Client libraries
> > don't have to change to continue using these CouchDB 4.0 endpoints
> > - This is the easiest to implement since it would re-use the
> > implementation for _changes feed (an extra option passed to the fold
> > function).
> > - Breaks API behavior if users relied on having a point-in-time[4]
> > snapshot view of the data.
> >
> > B) Simply end the stream. Let the users pass a `?transaction=true`
> > param which indicates they are aware the stream may end early and so
> > would have to paginate from the last emitted key with a skip=1. This
> > will keep the request bodies the same as current CouchDB. However, if
> > the users got all the data one request, they will end up wasting
> > another request to see if there is more data available. If they didn't
> > get any data they might have a too large of a skip value (see [2]) so
> > would have to guess different values for start/end keys. Or impose max
> > limit for the `skip` parameter.
> >
> > C) End the stream and add a final metadata row like a "transaction":
> > "timeout" at the end. That will let the user know to keep paginating
> > from the last key onward. This won't work for `_all_dbs` and
> > `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> > feeds and only use this for views and and _all_docs? If we like this
> > choice, let's think what happens for those as I couldn't come up with
> > anything decent there.
> >
> > D) Same as C but to solve the issue with skips[2], emit a bookmark
> > "key" of where the iteration stopped and the current "skip" and
> > "limit" params, which would keep decreasing. Then user would pass
> > those in "start_key=..." in the next request along with the limit and
> > skip params. So something like "continuation":{"skip":599, "limit":5,
> > "key":"..."}. This has the same issue with array results for
> > `_all_dbs` and `_dbs_info`[3].
> >
> > E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> > there such that response time is likely to fit in one transaction.
> > This could be tricky as different runtime environments will have
> > different characteristics. Also, if the timeout happens there isn't a
> > a nice way to send an HTTP error since we already sent the 200
> > response. The downside is that this might break how some users use the
> > API, if say the are using large skips and limits already. Perhaps here
> > we do both B and D, such that if users want transactional behavior,
> > they specify that `transaction=true` param and only then we enforce
> > low limit and skip maximums.
> >
> > F) At least for `_all_docs` it seems providing a point-in-time
> > snapshot view doesn't necessarily need to be tied to transaction
> > boundaries. We could check the update sequence of the database at the
> > start of the next transaction and if it hasn't changed we can continue
> > emitting a consistent view. This can apply to C and D and would just
> > determine when the stream ends. If there are no writes happening to
> > the db, this could potential streams all the data just like option A
> > would do. Not entirely sure if this would work for views.
> >
> > So what do we think? I can see different combinations of options here,
> > maybe even different for each API point. For example `_all_dbs`,
> > `_dbs_info` are always A, and `_all_docs` and views default to A but
> > have parameters to do F, etc.
> >
> > Cheers,
> > -Nick
> >
> > Some footnotes:
> >
> > [1] _changes feeds is the only one that works currently. It behaves as
> > per RFC
> > https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> > That is, we continue streaming the data by resetting the transaction
> > object and restarting from the last emitted key (db sequence in this
> > case). However, because the transaction restarts if a document is
> > updated while the streaming take place, it may appear in the _changes
> > feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> > have to document it, since previously we presented this point-in-time
> > snapshot of the database from when we started streaming.
> >
> > [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> > currently support efficient offsets for key selectors
> > (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> > we implemented skip by iterating over the data. This means that a skip
> > of say 100000 could keep timing out the transaction without yielding
> > any data.
> >
> > [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> > obvious place to insert a last metadata row.
> >
> > [4] For example they have a constraint that documents "a" and "z"
> > cannot both be in the database at the same time. But when iterating
> > it's possible that "a" was there at the start. Then by the end, "a"
> > was removed and "z" added, so both "a" and "z" would appear in the
> > emitted stream. Note that FoundationDB has APIs which exhibit the same
> > "relaxed" constrains:
> > https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
> >
Re: [DISCUSS] Streaming API in CouchDB 4.0
Posted by Mike Rhodes <co...@dx13.co.uk>.
Nick,
Thanks for thinking this through, it's certainly subtle and very unclear what is a "good" solution :(
I have a couple of thoughts, firstly about the guarantees we currently offer and then wondering whether there is an opportunity to improve our API by offering a single guarantee across all request types rather than bifurcating guarantees.
---
The first point is that, by my reasoning, CouchDB 2.x doesn't actually don't offer a point-in-time guarantee of the following sort currently. I read this as your saying Couch does offer this guarantee, apologies if I'm misreading:
> Document the API behavior change that it may
> present a view of the data is never a point-in-time[4] snapshot of the
> DB.
...
> [4] For example they have a constraint that documents "a" and "z"
> cannot both be in the database at the same time. But when iterating
> it's possible that "a" was there at the start. Then by the end, "a"
> was removed and "z" added, so both "a" and "z" would appear in the
> emitted stream. Note that FoundationDB has APIs which exhibit the same
> "relaxed" constrains:
> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
I don't believe we offer this guarantee because different database shards will respond to the scatter-gather inherent to a single global query type request at different times. This means that, given the following sequence of events:
(1) The shard containing "a" may start returning at time N.
(2) "a" may be deleted at N+1, but (1) will still be streaming from time N.
(3) "z" may be written to a second shard at time N+2.
(4) that second shard may not start returning until time N+3.
By my reasoning, "a" and "z" could thus appear in the same result set in current CouchDB, even if they never actually appear in the primary data at the same time (regardless of latency of shard replicas coming into agreement), voiding [4].
By my reckoning, you have point-in-time across a query request when you are working with a single shard, meaning we do have point in time for two scenarios:
- Partitioned queries.
- Q=1 databases.
Albeit this guarantee is still talking about the point in time of a single shard's replica rather than all replicas, meaning that further requests may produce different results if the shards are not in agreement. Which can perhaps be fixed by using stable=true.
I _think_ the working here is correct, but I'd welcome corrections in my understanding!
---
Our current behaviour seems extremely subtle and, I'd argue, unexpected. It is hard to reason about if you really need a particular guarantee.
Is there an opportunity to clarify behaviour here, such that we really _do_ guarantee point-in-time within _any_ single request, but only do this by leveraging FoundationDB's transaction isolation semantics and as such are only able to offer this based on the 5s timeout in place? The request boundary offers a very clear cut, user-visible boundary. This would obviously need to cover reads/writes of single docs and so on as well as probably needing further work w.r.t. bulk docs etc.
This restriction may naturally loosen as FoundationDB improves and the 5s timeout may be increased.
In this approach, my preference would be to add a closing line to the result stream to contain both a bookmark (based on the FoundationDB key perhaps rather than the index key of itself to avoid problems with skip/limit?) and a complete/not-complete boolean to enable clients to avoid the extra HTTP round-trip for completed result sets that Nick mentions.
---
For option (F), I feel that the "it sometimes works and sometimes doesn't" effect of checking the update-seq to see if we can continue streaming will be a confusing experience. I also find something similar with option (A) where a single request covers potentially many points in time and so feels hard to reason about, although it's a bit less subtle than today.
Footnote [2] seems quite a major problem, however, with the single transaction approach and as Nick says, it is hard to pick "good" maximums for skip -- perhaps users need to just avoid use of these in the new system given its behaviour? It feels like there's a definite "against the grain" aspect to these.
--
Mike.
On Wed, 19 Feb 2020, at 22:39, Nick Vatamaniuc wrote:
> Hello everyone,
>
> I'd like to discuss the shape and behavior of streaming APIs for CouchDB 4.x
>
> By "streaming APIs" I mean APIs which stream data in row as it gets
> read from the database. These are the endpoints I was thinking of:
>
> _all_docs, _all_dbs, _dbs_info and query results
>
> I want to focus on what happens when FoundationDB transactions
> time-out after 5 seconds. Currently, all those APIs except _changes[1]
> feeds, will crash or freeze. The reason is because the
> transaction_too_old error at the end of 5 seconds is retry-able by
> default, so the request handlers run again and end up shoving the
> whole request down the socket again, headers and all, which is
> obviously broken and not what we want.
>
> There are few alternatives discussed in couchdb-dev channel. I'll
> present some behaviors but feel free to add more. Some ideas might
> have been discounted on the IRC discussion already but I'll present
> them anyway in case is sparks further conversation:
>
> A) Do what _changes[1] feeds do. Start a new transaction and continue
> streaming the data from the next key after last emitted in the
> previous transaction. Document the API behavior change that it may
> present a view of the data is never a point-in-time[4] snapshot of the
> DB.
>
> - Keeps the API shape the same as CouchDB <4.0. Client libraries
> don't have to change to continue using these CouchDB 4.0 endpoints
> - This is the easiest to implement since it would re-use the
> implementation for _changes feed (an extra option passed to the fold
> function).
> - Breaks API behavior if users relied on having a point-in-time[4]
> snapshot view of the data.
>
> B) Simply end the stream. Let the users pass a `?transaction=true`
> param which indicates they are aware the stream may end early and so
> would have to paginate from the last emitted key with a skip=1. This
> will keep the request bodies the same as current CouchDB. However, if
> the users got all the data one request, they will end up wasting
> another request to see if there is more data available. If they didn't
> get any data they might have a too large of a skip value (see [2]) so
> would have to guess different values for start/end keys. Or impose max
> limit for the `skip` parameter.
>
> C) End the stream and add a final metadata row like a "transaction":
> "timeout" at the end. That will let the user know to keep paginating
> from the last key onward. This won't work for `_all_dbs` and
> `_dbs_info`[3] Maybe let those two endpoints behave like _changes
> feeds and only use this for views and and _all_docs? If we like this
> choice, let's think what happens for those as I couldn't come up with
> anything decent there.
>
> D) Same as C but to solve the issue with skips[2], emit a bookmark
> "key" of where the iteration stopped and the current "skip" and
> "limit" params, which would keep decreasing. Then user would pass
> those in "start_key=..." in the next request along with the limit and
> skip params. So something like "continuation":{"skip":599, "limit":5,
> "key":"..."}. This has the same issue with array results for
> `_all_dbs` and `_dbs_info`[3].
>
> E) Enforce low `limit` and `skip` parameters. Enforce maximum values
> there such that response time is likely to fit in one transaction.
> This could be tricky as different runtime environments will have
> different characteristics. Also, if the timeout happens there isn't a
> a nice way to send an HTTP error since we already sent the 200
> response. The downside is that this might break how some users use the
> API, if say the are using large skips and limits already. Perhaps here
> we do both B and D, such that if users want transactional behavior,
> they specify that `transaction=true` param and only then we enforce
> low limit and skip maximums.
>
> F) At least for `_all_docs` it seems providing a point-in-time
> snapshot view doesn't necessarily need to be tied to transaction
> boundaries. We could check the update sequence of the database at the
> start of the next transaction and if it hasn't changed we can continue
> emitting a consistent view. This can apply to C and D and would just
> determine when the stream ends. If there are no writes happening to
> the db, this could potential streams all the data just like option A
> would do. Not entirely sure if this would work for views.
>
> So what do we think? I can see different combinations of options here,
> maybe even different for each API point. For example `_all_dbs`,
> `_dbs_info` are always A, and `_all_docs` and views default to A but
> have parameters to do F, etc.
>
> Cheers,
> -Nick
>
> Some footnotes:
>
> [1] _changes feeds is the only one that works currently. It behaves as
> per RFC
> https://github.com/apache/couchdb-documentation/blob/master/rfcs/003-fdb-seq-index.md#access-patterns.
> That is, we continue streaming the data by resetting the transaction
> object and restarting from the last emitted key (db sequence in this
> case). However, because the transaction restarts if a document is
> updated while the streaming take place, it may appear in the _changes
> feed twice. That's a behavior difference from CouchDB < 4.0 and we'd
> have to document it, since previously we presented this point-in-time
> snapshot of the database from when we started streaming.
>
> [2] Our streaming APIs have both skips and limits. Since FDB doesn't
> currently support efficient offsets for key selectors
> (https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging)
> we implemented skip by iterating over the data. This means that a skip
> of say 100000 could keep timing out the transaction without yielding
> any data.
>
> [3] _all_dbs and _dbs_info return a JSON array so they don't have an
> obvious place to insert a last metadata row.
>
> [4] For example they have a constraint that documents "a" and "z"
> cannot both be in the database at the same time. But when iterating
> it's possible that "a" was there at the start. Then by the end, "a"
> was removed and "z" added, so both "a" and "z" would appear in the
> emitted stream. Note that FoundationDB has APIs which exhibit the same
> "relaxed" constrains:
> https://apple.github.io/foundationdb/api-python.html#module-fdb.locality
>