You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Garren Smith <ga...@apache.org> on 2019/05/16 06:51:28 UTC
Design doc index switching
Hi Everyone,
A common pattern we see for updating large indexes that can take a few days
to build, is create a new design docs with the new updated views. Then once
the new design doc is built, a user changes the new design doc’s id to the
old design doc. That way the CouchDB url for the views remain the same and
any requests to the design doc url automatically get the latest views only
once they built.
This is an effective way of managing building large indexes, but the
process is quite complicated and often users get it wrong. I would like to
propose that we move this process into CouchDB and let CouchDB handle the
actual process. From a users perspective, they would add a field to the
options of a design document that lets CouchDB know, that this build needs
to be built in the background and only replace the current index once its
built:
```
{
"_id": "_design/design-doc-id",
"_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
"views": {
"map-view": {
"map": "function (doc) {\n emit(doc._id, 1);\n}"
}
},
"language": "javascript",
"options": {
"build_and_replace": true
}
}
```
I think this is something we could build quite effectively once we have
CouchDB running on top of FoundationDB. I don’t want to implement it for
version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
we build out the map/reduce indexes.
What do you think? Any issues we might have by doing this internally?
Cheers
Garren
Re: Design doc index switching
Posted by ermouth <er...@gmail.com>.
Looks great, but how that shadow ddoc would replicate?
What happens when tgt node received shadow ddoc, rebuilt shadow index, and
then updates original ddoc?
ermouth
пт, 17 мая 2019 г. в 00:03, Robert Samuel Newson <rn...@apache.org>:
> I suggest an alternative; the new design document could include the _id of
> design document it’s replacing (“_replaces”:”_design/foo”). On completion
> of the view build of the new design document, CouchDB itself updates the
> named _id to the same content as the new design document (strictly, only
> the parts needed to make the view sig match) (perhaps it also deletes the
> new document).
>
>
Re: Design doc index switching
Posted by Garren Smith <ga...@apache.org>.
Stefan raises some really good points. Should we rather have an extra step
where the user can tell CouchDB when to do the switch over?
We could use the _info endpoint to indicate which design doc is currently
being used and which is being built. And even if the new design doc is
built.
Otherwise I also like the idea Bob proposes with the
(“_replaces”:”_design/foo”),
but maybe that is just an endpoint where a user can say replace ddoc1 with
ddoc2.
For both cases, the user would still have to monitor CouchDB to determine
when the view is fully built. So would this actually be helping?
On Fri, May 17, 2019 at 12:10 PM Jan Lehnardt <ja...@apache.org> wrote:
>
>
> > On 16. May 2019, at 23:03, Robert Samuel Newson <rn...@apache.org>
> wrote:
> >
> > I suggest an alternative; the new design document could include the _id
> of design document it’s replacing (“_replaces”:”_design/foo”). On
> completion of the view build of the new design document, CouchDB itself
> updates the named _id to the same content as the new design document
> (strictly, only the parts needed to make the view sig match) (perhaps it
> also deletes the new document).
> >
> > The advantage to this is that queries to the original design document
> continue to work throughout and at no point is there a discrepancy between
> the design documents contents (the map and reduce functions, etc) and the
> results you get from it.
>
>
> As Stefan points out, this only works if the consuming code knows how to
> handle both index result formats. If not, we need a way for either CouchDB
> to signal the build being ready or a way to confirm the swap to coincide
> with a code deploy.
>
>
> >
> > B.
> >
> >> On 16 May 2019, at 14:55, Jan Lehnardt <ja...@apache.org> wrote:
> >>
> >> +1 on solving this for all users, and same caveats as Stefan raises :)
> >>
> >>> On 16. May 2019, at 09:38, Stefan du Fresne <st...@medicmobile.org>
> wrote:
> >>>
> >>> Hey Garren,
> >>>
> >>> Having this a native part of CouchDB seems like a really cool idea: we
> have automated the manual dance you're talking about with our deployment
> tooling, but it would be really nice not to have to!
> >>>
> >>> I'm not clear how it would work though, at least in terms of coherent
> deployments. View changes are, like SQL migrations, an often non-backwards
> compatible change that has to occur as your new code deploys.
> >>>
> >>> Currently the naive approach is you deploy your new code alongside
> design doc changes, which then block view queries on first request until
> they're ready to go.
> >>>
> >>> The better approach is what you describe, which is what we do now,
> where we extract our design documents out of our deployment bundle and
> place them in a "staging" location to allow them to warm, then rename them
> and do the actual code deployment once that's complete (managed by an
> external deployment service we built). This importantly lets us split the
> "warming" bit from the deployment bit: we only deploy new code once the
> design documents that are shipped with that code is ready to go.
> >>>
> >>> How would you foresee this kind of flow happening here? Would there be
> a way to query the design doc to know if it had flipped to the new version
> yet? Would you be able to control when this flip occurs? Or would the
> expectation be that your code handles both versions gracefully?
> >>>
> >>> As an example to mull over, let's say you have design doc v1, which
> has view a. You push design doc v2, which has added view b, but has also
> changed view a in some backwards incompatible way. While v2 is still
> building and is not yet the active doc:
> >>> - If you queried view a you'd get the v1 version, that's clear
> >>> - If you queried view b you'd get... a 404? Some other custom code?
> >>> - If you GET the design document what doc would you see? Presumably v2?
> >>> - Could you query something to determine which version is currently
> active? Or perhaps just whether there is a background version building at
> all?
> >>>
> >>> Cheers,
> >>> Stefan
> >>>
> >>>> On 16 May 2019, at 07:51, Garren Smith <ga...@apache.org> wrote:
> >>>>
> >>>> Hi Everyone,
> >>>>
> >>>> A common pattern we see for updating large indexes that can take a
> few days
> >>>> to build, is create a new design docs with the new updated views.
> Then once
> >>>> the new design doc is built, a user changes the new design doc’s id
> to the
> >>>> old design doc. That way the CouchDB url for the views remain the
> same and
> >>>> any requests to the design doc url automatically get the latest views
> only
> >>>> once they built.
> >>>>
> >>>> This is an effective way of managing building large indexes, but the
> >>>> process is quite complicated and often users get it wrong. I would
> like to
> >>>> propose that we move this process into CouchDB and let CouchDB handle
> the
> >>>> actual process. From a users perspective, they would add a field to
> the
> >>>> options of a design document that lets CouchDB know, that this build
> needs
> >>>> to be built in the background and only replace the current index once
> its
> >>>> built:
> >>>>
> >>>> ```
> >>>> {
> >>>> "_id": "_design/design-doc-id",
> >>>> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
> >>>> "views": {
> >>>> "map-view": {
> >>>> "map": "function (doc) {\n emit(doc._id, 1);\n}"
> >>>> }
> >>>> },
> >>>> "language": "javascript",
> >>>> "options": {
> >>>> "build_and_replace": true
> >>>> }
> >>>> }
> >>>> ```
> >>>>
> >>>> I think this is something we could build quite effectively once we
> have
> >>>> CouchDB running on top of FoundationDB. I don’t want to implement it
> for
> >>>> version 1 of CouchDB on FDB, but it would be nice to keep this in
> mind as
> >>>> we build out the map/reduce indexes.
> >>>>
> >>>> What do you think? Any issues we might have by doing this internally?
> >>>>
> >>>> Cheers
> >>>> Garren
> >>>
> >>
> >> --
> >> Professional Support for Apache CouchDB:
> >> https://neighbourhood.ie/couchdb-support/
> >>
> >
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
>
Re: Design doc index switching
Posted by Mike Rhodes <co...@dx13.co.uk>.
From all the messages in this thread, it sounds like there are two basic forms of change to indexes here:
- Backwards compatible change (e.g., adding a field to the emitted value where the consumer is agnostic to new fields).
- Backwards incompatible changes.
I think the automatic replace works well in the first case, but badly in the second. Because of the fact that any backwards incompatible change requires an application deployment by definition, and so having auto-flip of the output of the index happening would be a road to either over complicated application code to handle both cases or just the application being down for a time.
TBH if I were doing the second case, I'd be building entirely new indexes and just switching my code to use them, then dropping the old indexes. That is, creating a new ddoc, letting that index build, moving my application to that new index. The new ddoc is because of view groups, of course.
I wonder whether this question is simpler if viewgroups are removed as a concept, such that if I change an index's definition, only that index is rebuilt as opposed to other indexes that I put in the same viewgroup. Certainly, the manual/auto-background-rebuild thing seems to be mostly motivated by people making changes to one index (or adding a new index) in a view group and not wanting all the other indexes to be out of action during the new index's build process.
I'm mostly wondering whether this is a workaround to a more fundamental design decision?
--
Mike.
On Fri, 17 May 2019, at 14:26, Robert Samuel Newson wrote:
> If the consuming code can’t handle the switchover, the user should not
> be asking couchdb to build an incompatible index in the first place,
> but certainly shouldn’t ask couchdb to replace an index automatically
> (i.e, they should not use our proposed new handling at all).
>
> On replication, the doc would replicate as normal, causing the new
> index to build wherever it lands. Each couchdb instance would do the
> swap over independently. To address conflicts, we could block edits to
> any design doc with a _replaces item if the view is still building.
> Like for replication docs, and similarly we’d only allow a delete,
> which would delete the new index also.
>
> B.
>
> > On 17 May 2019, at 10:59, Jan Lehnardt <ja...@apache.org> wrote:
> >
> >
> >
> >> On 16. May 2019, at 23:03, Robert Samuel Newson <rn...@apache.org> wrote:
> >>
> >> I suggest an alternative; the new design document could include the _id of design document it’s replacing (“_replaces”:”_design/foo”). On completion of the view build of the new design document, CouchDB itself updates the named _id to the same content as the new design document (strictly, only the parts needed to make the view sig match) (perhaps it also deletes the new document).
> >>
> >> The advantage to this is that queries to the original design document continue to work throughout and at no point is there a discrepancy between the design documents contents (the map and reduce functions, etc) and the results you get from it.
> >
> >
> > As Stefan points out, this only works if the consuming code knows how to handle both index result formats. If not, we need a way for either CouchDB to signal the build being ready or a way to confirm the swap to coincide with a code deploy.
> >
> >
> >>
> >> B.
> >>
> >>> On 16 May 2019, at 14:55, Jan Lehnardt <ja...@apache.org> wrote:
> >>>
> >>> +1 on solving this for all users, and same caveats as Stefan raises :)
> >>>
> >>>> On 16. May 2019, at 09:38, Stefan du Fresne <st...@medicmobile.org> wrote:
> >>>>
> >>>> Hey Garren,
> >>>>
> >>>> Having this a native part of CouchDB seems like a really cool idea: we have automated the manual dance you're talking about with our deployment tooling, but it would be really nice not to have to!
> >>>>
> >>>> I'm not clear how it would work though, at least in terms of coherent deployments. View changes are, like SQL migrations, an often non-backwards compatible change that has to occur as your new code deploys.
> >>>>
> >>>> Currently the naive approach is you deploy your new code alongside design doc changes, which then block view queries on first request until they're ready to go.
> >>>>
> >>>> The better approach is what you describe, which is what we do now, where we extract our design documents out of our deployment bundle and place them in a "staging" location to allow them to warm, then rename them and do the actual code deployment once that's complete (managed by an external deployment service we built). This importantly lets us split the "warming" bit from the deployment bit: we only deploy new code once the design documents that are shipped with that code is ready to go.
> >>>>
> >>>> How would you foresee this kind of flow happening here? Would there be a way to query the design doc to know if it had flipped to the new version yet? Would you be able to control when this flip occurs? Or would the expectation be that your code handles both versions gracefully?
> >>>>
> >>>> As an example to mull over, let's say you have design doc v1, which has view a. You push design doc v2, which has added view b, but has also changed view a in some backwards incompatible way. While v2 is still building and is not yet the active doc:
> >>>> - If you queried view a you'd get the v1 version, that's clear
> >>>> - If you queried view b you'd get... a 404? Some other custom code?
> >>>> - If you GET the design document what doc would you see? Presumably v2?
> >>>> - Could you query something to determine which version is currently active? Or perhaps just whether there is a background version building at all?
> >>>>
> >>>> Cheers,
> >>>> Stefan
> >>>>
> >>>>> On 16 May 2019, at 07:51, Garren Smith <ga...@apache.org> wrote:
> >>>>>
> >>>>> Hi Everyone,
> >>>>>
> >>>>> A common pattern we see for updating large indexes that can take a few days
> >>>>> to build, is create a new design docs with the new updated views. Then once
> >>>>> the new design doc is built, a user changes the new design doc’s id to the
> >>>>> old design doc. That way the CouchDB url for the views remain the same and
> >>>>> any requests to the design doc url automatically get the latest views only
> >>>>> once they built.
> >>>>>
> >>>>> This is an effective way of managing building large indexes, but the
> >>>>> process is quite complicated and often users get it wrong. I would like to
> >>>>> propose that we move this process into CouchDB and let CouchDB handle the
> >>>>> actual process. From a users perspective, they would add a field to the
> >>>>> options of a design document that lets CouchDB know, that this build needs
> >>>>> to be built in the background and only replace the current index once its
> >>>>> built:
> >>>>>
> >>>>> ```
> >>>>> {
> >>>>> "_id": "_design/design-doc-id",
> >>>>> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
> >>>>> "views": {
> >>>>> "map-view": {
> >>>>> "map": "function (doc) {\n emit(doc._id, 1);\n}"
> >>>>> }
> >>>>> },
> >>>>> "language": "javascript",
> >>>>> "options": {
> >>>>> "build_and_replace": true
> >>>>> }
> >>>>> }
> >>>>> ```
> >>>>>
> >>>>> I think this is something we could build quite effectively once we have
> >>>>> CouchDB running on top of FoundationDB. I don’t want to implement it for
> >>>>> version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
> >>>>> we build out the map/reduce indexes.
> >>>>>
> >>>>> What do you think? Any issues we might have by doing this internally?
> >>>>>
> >>>>> Cheers
> >>>>> Garren
> >>>>
> >>>
> >>> --
> >>> Professional Support for Apache CouchDB:
> >>> https://neighbourhood.ie/couchdb-support/
> >>>
> >>
> >
> > --
> > Professional Support for Apache CouchDB:
> > https://neighbourhood.ie/couchdb-support/
>
>
Re: Design doc index switching
Posted by Robert Samuel Newson <rn...@apache.org>.
If the consuming code can’t handle the switchover, the user should not be asking couchdb to build an incompatible index in the first place, but certainly shouldn’t ask couchdb to replace an index automatically (i.e, they should not use our proposed new handling at all).
On replication, the doc would replicate as normal, causing the new index to build wherever it lands. Each couchdb instance would do the swap over independently. To address conflicts, we could block edits to any design doc with a _replaces item if the view is still building. Like for replication docs, and similarly we’d only allow a delete, which would delete the new index also.
B.
> On 17 May 2019, at 10:59, Jan Lehnardt <ja...@apache.org> wrote:
>
>
>
>> On 16. May 2019, at 23:03, Robert Samuel Newson <rn...@apache.org> wrote:
>>
>> I suggest an alternative; the new design document could include the _id of design document it’s replacing (“_replaces”:”_design/foo”). On completion of the view build of the new design document, CouchDB itself updates the named _id to the same content as the new design document (strictly, only the parts needed to make the view sig match) (perhaps it also deletes the new document).
>>
>> The advantage to this is that queries to the original design document continue to work throughout and at no point is there a discrepancy between the design documents contents (the map and reduce functions, etc) and the results you get from it.
>
>
> As Stefan points out, this only works if the consuming code knows how to handle both index result formats. If not, we need a way for either CouchDB to signal the build being ready or a way to confirm the swap to coincide with a code deploy.
>
>
>>
>> B.
>>
>>> On 16 May 2019, at 14:55, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> +1 on solving this for all users, and same caveats as Stefan raises :)
>>>
>>>> On 16. May 2019, at 09:38, Stefan du Fresne <st...@medicmobile.org> wrote:
>>>>
>>>> Hey Garren,
>>>>
>>>> Having this a native part of CouchDB seems like a really cool idea: we have automated the manual dance you're talking about with our deployment tooling, but it would be really nice not to have to!
>>>>
>>>> I'm not clear how it would work though, at least in terms of coherent deployments. View changes are, like SQL migrations, an often non-backwards compatible change that has to occur as your new code deploys.
>>>>
>>>> Currently the naive approach is you deploy your new code alongside design doc changes, which then block view queries on first request until they're ready to go.
>>>>
>>>> The better approach is what you describe, which is what we do now, where we extract our design documents out of our deployment bundle and place them in a "staging" location to allow them to warm, then rename them and do the actual code deployment once that's complete (managed by an external deployment service we built). This importantly lets us split the "warming" bit from the deployment bit: we only deploy new code once the design documents that are shipped with that code is ready to go.
>>>>
>>>> How would you foresee this kind of flow happening here? Would there be a way to query the design doc to know if it had flipped to the new version yet? Would you be able to control when this flip occurs? Or would the expectation be that your code handles both versions gracefully?
>>>>
>>>> As an example to mull over, let's say you have design doc v1, which has view a. You push design doc v2, which has added view b, but has also changed view a in some backwards incompatible way. While v2 is still building and is not yet the active doc:
>>>> - If you queried view a you'd get the v1 version, that's clear
>>>> - If you queried view b you'd get... a 404? Some other custom code?
>>>> - If you GET the design document what doc would you see? Presumably v2?
>>>> - Could you query something to determine which version is currently active? Or perhaps just whether there is a background version building at all?
>>>>
>>>> Cheers,
>>>> Stefan
>>>>
>>>>> On 16 May 2019, at 07:51, Garren Smith <ga...@apache.org> wrote:
>>>>>
>>>>> Hi Everyone,
>>>>>
>>>>> A common pattern we see for updating large indexes that can take a few days
>>>>> to build, is create a new design docs with the new updated views. Then once
>>>>> the new design doc is built, a user changes the new design doc’s id to the
>>>>> old design doc. That way the CouchDB url for the views remain the same and
>>>>> any requests to the design doc url automatically get the latest views only
>>>>> once they built.
>>>>>
>>>>> This is an effective way of managing building large indexes, but the
>>>>> process is quite complicated and often users get it wrong. I would like to
>>>>> propose that we move this process into CouchDB and let CouchDB handle the
>>>>> actual process. From a users perspective, they would add a field to the
>>>>> options of a design document that lets CouchDB know, that this build needs
>>>>> to be built in the background and only replace the current index once its
>>>>> built:
>>>>>
>>>>> ```
>>>>> {
>>>>> "_id": "_design/design-doc-id",
>>>>> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
>>>>> "views": {
>>>>> "map-view": {
>>>>> "map": "function (doc) {\n emit(doc._id, 1);\n}"
>>>>> }
>>>>> },
>>>>> "language": "javascript",
>>>>> "options": {
>>>>> "build_and_replace": true
>>>>> }
>>>>> }
>>>>> ```
>>>>>
>>>>> I think this is something we could build quite effectively once we have
>>>>> CouchDB running on top of FoundationDB. I don’t want to implement it for
>>>>> version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
>>>>> we build out the map/reduce indexes.
>>>>>
>>>>> What do you think? Any issues we might have by doing this internally?
>>>>>
>>>>> Cheers
>>>>> Garren
>>>>
>>>
>>> --
>>> Professional Support for Apache CouchDB:
>>> https://neighbourhood.ie/couchdb-support/
>>>
>>
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
Re: Design doc index switching
Posted by Jan Lehnardt <ja...@apache.org>.
> On 16. May 2019, at 23:03, Robert Samuel Newson <rn...@apache.org> wrote:
>
> I suggest an alternative; the new design document could include the _id of design document it’s replacing (“_replaces”:”_design/foo”). On completion of the view build of the new design document, CouchDB itself updates the named _id to the same content as the new design document (strictly, only the parts needed to make the view sig match) (perhaps it also deletes the new document).
>
> The advantage to this is that queries to the original design document continue to work throughout and at no point is there a discrepancy between the design documents contents (the map and reduce functions, etc) and the results you get from it.
As Stefan points out, this only works if the consuming code knows how to handle both index result formats. If not, we need a way for either CouchDB to signal the build being ready or a way to confirm the swap to coincide with a code deploy.
>
> B.
>
>> On 16 May 2019, at 14:55, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> +1 on solving this for all users, and same caveats as Stefan raises :)
>>
>>> On 16. May 2019, at 09:38, Stefan du Fresne <st...@medicmobile.org> wrote:
>>>
>>> Hey Garren,
>>>
>>> Having this a native part of CouchDB seems like a really cool idea: we have automated the manual dance you're talking about with our deployment tooling, but it would be really nice not to have to!
>>>
>>> I'm not clear how it would work though, at least in terms of coherent deployments. View changes are, like SQL migrations, an often non-backwards compatible change that has to occur as your new code deploys.
>>>
>>> Currently the naive approach is you deploy your new code alongside design doc changes, which then block view queries on first request until they're ready to go.
>>>
>>> The better approach is what you describe, which is what we do now, where we extract our design documents out of our deployment bundle and place them in a "staging" location to allow them to warm, then rename them and do the actual code deployment once that's complete (managed by an external deployment service we built). This importantly lets us split the "warming" bit from the deployment bit: we only deploy new code once the design documents that are shipped with that code is ready to go.
>>>
>>> How would you foresee this kind of flow happening here? Would there be a way to query the design doc to know if it had flipped to the new version yet? Would you be able to control when this flip occurs? Or would the expectation be that your code handles both versions gracefully?
>>>
>>> As an example to mull over, let's say you have design doc v1, which has view a. You push design doc v2, which has added view b, but has also changed view a in some backwards incompatible way. While v2 is still building and is not yet the active doc:
>>> - If you queried view a you'd get the v1 version, that's clear
>>> - If you queried view b you'd get... a 404? Some other custom code?
>>> - If you GET the design document what doc would you see? Presumably v2?
>>> - Could you query something to determine which version is currently active? Or perhaps just whether there is a background version building at all?
>>>
>>> Cheers,
>>> Stefan
>>>
>>>> On 16 May 2019, at 07:51, Garren Smith <ga...@apache.org> wrote:
>>>>
>>>> Hi Everyone,
>>>>
>>>> A common pattern we see for updating large indexes that can take a few days
>>>> to build, is create a new design docs with the new updated views. Then once
>>>> the new design doc is built, a user changes the new design doc’s id to the
>>>> old design doc. That way the CouchDB url for the views remain the same and
>>>> any requests to the design doc url automatically get the latest views only
>>>> once they built.
>>>>
>>>> This is an effective way of managing building large indexes, but the
>>>> process is quite complicated and often users get it wrong. I would like to
>>>> propose that we move this process into CouchDB and let CouchDB handle the
>>>> actual process. From a users perspective, they would add a field to the
>>>> options of a design document that lets CouchDB know, that this build needs
>>>> to be built in the background and only replace the current index once its
>>>> built:
>>>>
>>>> ```
>>>> {
>>>> "_id": "_design/design-doc-id",
>>>> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
>>>> "views": {
>>>> "map-view": {
>>>> "map": "function (doc) {\n emit(doc._id, 1);\n}"
>>>> }
>>>> },
>>>> "language": "javascript",
>>>> "options": {
>>>> "build_and_replace": true
>>>> }
>>>> }
>>>> ```
>>>>
>>>> I think this is something we could build quite effectively once we have
>>>> CouchDB running on top of FoundationDB. I don’t want to implement it for
>>>> version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
>>>> we build out the map/reduce indexes.
>>>>
>>>> What do you think? Any issues we might have by doing this internally?
>>>>
>>>> Cheers
>>>> Garren
>>>
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: Design doc index switching
Posted by Robert Samuel Newson <rn...@apache.org>.
I suggest an alternative; the new design document could include the _id of design document it’s replacing (“_replaces”:”_design/foo”). On completion of the view build of the new design document, CouchDB itself updates the named _id to the same content as the new design document (strictly, only the parts needed to make the view sig match) (perhaps it also deletes the new document).
The advantage to this is that queries to the original design document continue to work throughout and at no point is there a discrepancy between the design documents contents (the map and reduce functions, etc) and the results you get from it.
B.
> On 16 May 2019, at 14:55, Jan Lehnardt <ja...@apache.org> wrote:
>
> +1 on solving this for all users, and same caveats as Stefan raises :)
>
>> On 16. May 2019, at 09:38, Stefan du Fresne <st...@medicmobile.org> wrote:
>>
>> Hey Garren,
>>
>> Having this a native part of CouchDB seems like a really cool idea: we have automated the manual dance you're talking about with our deployment tooling, but it would be really nice not to have to!
>>
>> I'm not clear how it would work though, at least in terms of coherent deployments. View changes are, like SQL migrations, an often non-backwards compatible change that has to occur as your new code deploys.
>>
>> Currently the naive approach is you deploy your new code alongside design doc changes, which then block view queries on first request until they're ready to go.
>>
>> The better approach is what you describe, which is what we do now, where we extract our design documents out of our deployment bundle and place them in a "staging" location to allow them to warm, then rename them and do the actual code deployment once that's complete (managed by an external deployment service we built). This importantly lets us split the "warming" bit from the deployment bit: we only deploy new code once the design documents that are shipped with that code is ready to go.
>>
>> How would you foresee this kind of flow happening here? Would there be a way to query the design doc to know if it had flipped to the new version yet? Would you be able to control when this flip occurs? Or would the expectation be that your code handles both versions gracefully?
>>
>> As an example to mull over, let's say you have design doc v1, which has view a. You push design doc v2, which has added view b, but has also changed view a in some backwards incompatible way. While v2 is still building and is not yet the active doc:
>> - If you queried view a you'd get the v1 version, that's clear
>> - If you queried view b you'd get... a 404? Some other custom code?
>> - If you GET the design document what doc would you see? Presumably v2?
>> - Could you query something to determine which version is currently active? Or perhaps just whether there is a background version building at all?
>>
>> Cheers,
>> Stefan
>>
>>> On 16 May 2019, at 07:51, Garren Smith <ga...@apache.org> wrote:
>>>
>>> Hi Everyone,
>>>
>>> A common pattern we see for updating large indexes that can take a few days
>>> to build, is create a new design docs with the new updated views. Then once
>>> the new design doc is built, a user changes the new design doc’s id to the
>>> old design doc. That way the CouchDB url for the views remain the same and
>>> any requests to the design doc url automatically get the latest views only
>>> once they built.
>>>
>>> This is an effective way of managing building large indexes, but the
>>> process is quite complicated and often users get it wrong. I would like to
>>> propose that we move this process into CouchDB and let CouchDB handle the
>>> actual process. From a users perspective, they would add a field to the
>>> options of a design document that lets CouchDB know, that this build needs
>>> to be built in the background and only replace the current index once its
>>> built:
>>>
>>> ```
>>> {
>>> "_id": "_design/design-doc-id",
>>> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
>>> "views": {
>>> "map-view": {
>>> "map": "function (doc) {\n emit(doc._id, 1);\n}"
>>> }
>>> },
>>> "language": "javascript",
>>> "options": {
>>> "build_and_replace": true
>>> }
>>> }
>>> ```
>>>
>>> I think this is something we could build quite effectively once we have
>>> CouchDB running on top of FoundationDB. I don’t want to implement it for
>>> version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
>>> we build out the map/reduce indexes.
>>>
>>> What do you think? Any issues we might have by doing this internally?
>>>
>>> Cheers
>>> Garren
>>
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
Re: Design doc index switching
Posted by Jan Lehnardt <ja...@apache.org>.
+1 on solving this for all users, and same caveats as Stefan raises :)
> On 16. May 2019, at 09:38, Stefan du Fresne <st...@medicmobile.org> wrote:
>
> Hey Garren,
>
> Having this a native part of CouchDB seems like a really cool idea: we have automated the manual dance you're talking about with our deployment tooling, but it would be really nice not to have to!
>
> I'm not clear how it would work though, at least in terms of coherent deployments. View changes are, like SQL migrations, an often non-backwards compatible change that has to occur as your new code deploys.
>
> Currently the naive approach is you deploy your new code alongside design doc changes, which then block view queries on first request until they're ready to go.
>
> The better approach is what you describe, which is what we do now, where we extract our design documents out of our deployment bundle and place them in a "staging" location to allow them to warm, then rename them and do the actual code deployment once that's complete (managed by an external deployment service we built). This importantly lets us split the "warming" bit from the deployment bit: we only deploy new code once the design documents that are shipped with that code is ready to go.
>
> How would you foresee this kind of flow happening here? Would there be a way to query the design doc to know if it had flipped to the new version yet? Would you be able to control when this flip occurs? Or would the expectation be that your code handles both versions gracefully?
>
> As an example to mull over, let's say you have design doc v1, which has view a. You push design doc v2, which has added view b, but has also changed view a in some backwards incompatible way. While v2 is still building and is not yet the active doc:
> - If you queried view a you'd get the v1 version, that's clear
> - If you queried view b you'd get... a 404? Some other custom code?
> - If you GET the design document what doc would you see? Presumably v2?
> - Could you query something to determine which version is currently active? Or perhaps just whether there is a background version building at all?
>
> Cheers,
> Stefan
>
>> On 16 May 2019, at 07:51, Garren Smith <ga...@apache.org> wrote:
>>
>> Hi Everyone,
>>
>> A common pattern we see for updating large indexes that can take a few days
>> to build, is create a new design docs with the new updated views. Then once
>> the new design doc is built, a user changes the new design doc’s id to the
>> old design doc. That way the CouchDB url for the views remain the same and
>> any requests to the design doc url automatically get the latest views only
>> once they built.
>>
>> This is an effective way of managing building large indexes, but the
>> process is quite complicated and often users get it wrong. I would like to
>> propose that we move this process into CouchDB and let CouchDB handle the
>> actual process. From a users perspective, they would add a field to the
>> options of a design document that lets CouchDB know, that this build needs
>> to be built in the background and only replace the current index once its
>> built:
>>
>> ```
>> {
>> "_id": "_design/design-doc-id",
>> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
>> "views": {
>> "map-view": {
>> "map": "function (doc) {\n emit(doc._id, 1);\n}"
>> }
>> },
>> "language": "javascript",
>> "options": {
>> "build_and_replace": true
>> }
>> }
>> ```
>>
>> I think this is something we could build quite effectively once we have
>> CouchDB running on top of FoundationDB. I don’t want to implement it for
>> version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
>> we build out the map/reduce indexes.
>>
>> What do you think? Any issues we might have by doing this internally?
>>
>> Cheers
>> Garren
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: Design doc index switching
Posted by Stefan du Fresne <st...@medicmobile.org>.
Hey Garren,
Having this a native part of CouchDB seems like a really cool idea: we have automated the manual dance you're talking about with our deployment tooling, but it would be really nice not to have to!
I'm not clear how it would work though, at least in terms of coherent deployments. View changes are, like SQL migrations, an often non-backwards compatible change that has to occur as your new code deploys.
Currently the naive approach is you deploy your new code alongside design doc changes, which then block view queries on first request until they're ready to go.
The better approach is what you describe, which is what we do now, where we extract our design documents out of our deployment bundle and place them in a "staging" location to allow them to warm, then rename them and do the actual code deployment once that's complete (managed by an external deployment service we built). This importantly lets us split the "warming" bit from the deployment bit: we only deploy new code once the design documents that are shipped with that code is ready to go.
How would you foresee this kind of flow happening here? Would there be a way to query the design doc to know if it had flipped to the new version yet? Would you be able to control when this flip occurs? Or would the expectation be that your code handles both versions gracefully?
As an example to mull over, let's say you have design doc v1, which has view a. You push design doc v2, which has added view b, but has also changed view a in some backwards incompatible way. While v2 is still building and is not yet the active doc:
- If you queried view a you'd get the v1 version, that's clear
- If you queried view b you'd get... a 404? Some other custom code?
- If you GET the design document what doc would you see? Presumably v2?
- Could you query something to determine which version is currently active? Or perhaps just whether there is a background version building at all?
Cheers,
Stefan
> On 16 May 2019, at 07:51, Garren Smith <ga...@apache.org> wrote:
>
> Hi Everyone,
>
> A common pattern we see for updating large indexes that can take a few days
> to build, is create a new design docs with the new updated views. Then once
> the new design doc is built, a user changes the new design doc’s id to the
> old design doc. That way the CouchDB url for the views remain the same and
> any requests to the design doc url automatically get the latest views only
> once they built.
>
> This is an effective way of managing building large indexes, but the
> process is quite complicated and often users get it wrong. I would like to
> propose that we move this process into CouchDB and let CouchDB handle the
> actual process. From a users perspective, they would add a field to the
> options of a design document that lets CouchDB know, that this build needs
> to be built in the background and only replace the current index once its
> built:
>
> ```
> {
> "_id": "_design/design-doc-id",
> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
> "views": {
> "map-view": {
> "map": "function (doc) {\n emit(doc._id, 1);\n}"
> }
> },
> "language": "javascript",
> "options": {
> "build_and_replace": true
> }
> }
> ```
>
> I think this is something we could build quite effectively once we have
> CouchDB running on top of FoundationDB. I don’t want to implement it for
> version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
> we build out the map/reduce indexes.
>
> What do you think? Any issues we might have by doing this internally?
>
> Cheers
> Garren
Re: Design doc index switching
Posted by Joan Touzet <wo...@apache.org>.
Hi Garren,
+1. I actually went hunting in GitHub for an issue on this, and can't
find one. It probably goes back to JIRA, and I don't have the energy to
dig through that now.
The closest issue that captures this is the same thing - but for
*databases* - and is on our official roadmap from the last summit:
https://github.com/apache/couchdb/issues/1502
Could we consider this use case at the same time?
-Joan
On 2019-05-16 2:51 a.m., Garren Smith wrote:
> Hi Everyone,
>
> A common pattern we see for updating large indexes that can take a few days
> to build, is create a new design docs with the new updated views. Then once
> the new design doc is built, a user changes the new design doc’s id to the
> old design doc. That way the CouchDB url for the views remain the same and
> any requests to the design doc url automatically get the latest views only
> once they built.
>
> This is an effective way of managing building large indexes, but the
> process is quite complicated and often users get it wrong. I would like to
> propose that we move this process into CouchDB and let CouchDB handle the
> actual process. From a users perspective, they would add a field to the
> options of a design document that lets CouchDB know, that this build needs
> to be built in the background and only replace the current index once its
> built:
>
> ```
> {
> "_id": "_design/design-doc-id",
> "_rev": "2-8d361a23b4cb8e213f0868ea3d2742c2",
> "views": {
> "map-view": {
> "map": "function (doc) {\n emit(doc._id, 1);\n}"
> }
> },
> "language": "javascript",
> "options": {
> "build_and_replace": true
> }
> }
> ```
>
> I think this is something we could build quite effectively once we have
> CouchDB running on top of FoundationDB. I don’t want to implement it for
> version 1 of CouchDB on FDB, but it would be nice to keep this in mind as
> we build out the map/reduce indexes.
>
> What do you think? Any issues we might have by doing this internally?
>
> Cheers
> Garren
>