Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc <va...@gmail.com>]

Hi everyone,

The pull request is open and ready for review:
https://github.com/apache/couchdb/pull/3889

In summary, I think we managed to track the collation versions in
views in a backwards compatible manner. There is a new view info map
in the header which can be used in the future to record additional
metadata about views (couchjs versions?, etc). With a slight bit of
trickery, the views can also be transparently downgraded to the 3.2.1
version without a signature change or a view reset.

Views which have more than one collator version will be submitted for
re-compaction to the already existing smoosh upgrade views channel.
The automatic trigger can be disabled as well with a new config
option.

As far as HTTP API differences these two endpoints now emit additional
collator version info:
  _design/*/_info "view_index" has a "collator_versions" list which
shows the list of versions for that particular view
  _node/*/_versions "collation_driver" object has a new
"collator_version" which shows the collator version
An example of this can be seen in
https://github.com/apache/couchdb/pull/3889#issuecomment-1022643789

Thanks to everyone who participated in the discussion! Let's continue
the review in the PR comments

Cheers,
-Nick

On Tue, Jan 25, 2022 at 10:15 AM Nick Vatamaniuc <va...@gmail.com> wrote:
>
> Good idea, Will, to return the current collator version in the
> `/_node/_local/_versions` output. We return the collation algorithm
> and the library versions, however, since we switched to tracking the
> opaque collator version, it's good to also expose that too.
>
> On Tue, Jan 25, 2022 at 9:47 AM Will Young <lo...@gmail.com> wrote:
> >
> > Hi Nick,
> >
> >   The view _info setup looks good to me. Maybe it would be helpful to
> > print the current runtime's collator and icu versions somewhere like
> > the / meta or  /node/ _system endpoint? I think that would provide a
> > way to cross-reference to alleviate the drawback of the collator being
> > the least human readable version (though only to find a more readable
> > version for the views that are from the current runtime,) and maybe to
> > debug oddities like a cluster somehow having nodes that are out of
> > sync on libicu versions, or just to make it easier to check if dbs are
> > going to be rebuilt after an update. Of course there are also other
> > ways for an admin to examine the current runtime and workout versions
> > so it is probably a question of how frequently it will come up.
> >
> > Thanks,
> > Will
> >
> > Am Di., 25. Jan. 2022 um 07:45 Uhr schrieb Nick Vatamaniuc <va...@gmail.com>:
> > >
> > > Another update regarding the draft PR.
> > >
> > > There are now upgrade tests to see how we handle older 2.x, 3.2.1, and
> > > views with multiple collator versions in them.
> > >
> > > The last commit modifies the _design/*/_info API to return the list of
> > > collator versions to the user and wanted to see what everyone thought
> > > about it:
> > >
> > > https://github.com/apache/couchdb/pull/3889#issuecomment-1020861208
> > >
> > > Thanks,
> > > -Nick
> > >
> > > On Tue, Jan 11, 2022 at 1:06 PM Nick Vatamaniuc <va...@gmail.com> wrote:
> > > >
> > > > I threw together a draft PR https://github.com/apache/couchdb/pull/3889
> > > >
> > > > Would that work? There are two tricks there - re-using a field
> > > > position from an older <2.3.1 format, this should allow transparently
> > > > downgrading back to 3.2.1 as we ignore that field there. Also, used a
> > > > map  so it should allow adding extra info to the views in the future
> > > > (custom collation tailorings?).
> > > >
> > > > Thanks,
> > > > -Nick
> > > >
> > > > On Sat, Nov 20, 2021 at 12:32 PM Nick Vatamaniuc <va...@gmail.com> wrote:
> > > > >
> > > > > Thanks, Adam. And thanks for the tip about the view header, Bob.
> > > > >
> > > > > Wonder if a disk version would make sense for views. Noticed Eric did
> > > > > a nice job transparently migrating 2.x -> 3.x view files when we
> > > > > removed key seq indices. Perhaps something like that would work for
> > > > > adding a collator version.
> > > > >
> > > > > Cheers,
> > > > > -Nick
> > > > >
> > > > > On Fri, Nov 19, 2021 at 9:09 AM Adam Kocoloski <ko...@apache.org> wrote:
> > > > > >
> > > > > > That seems like a smart solution Nick.
> > > > > >
> > > > > > Adam
> > > > > >
> > > > > > > On Nov 19, 2021, at 7:28 AM, Robert Newson <bo...@rsn.io> wrote:
> > > > > > >
> > > > > > > Noting that the upgrade channel for views was misconceived (by me) as there is no version number in the header for them. You’d need to add it.
> > > > > > >
> > > > > > > B.
> > > > > > >
> > > > > > >> On 18 Nov 2021, at 07:12, Nick Vatamaniuc <va...@gmail.com> wrote:
> > > > > > >>
> > > > > > >> Thinking more about this issue I wonder if we can avoid resetting and
> > > > > > >> rebuilding everything from scratch, and instead, let the upgrade
> > > > > > >> happen in the background, while still serving the existing view data.
> > > > > > >>
> > > > > > >> The realization was that collation doesn't affect the emitted keys and
> > > > > > >> values themselves, only their order in the view b-trees. That means
> > > > > > >> we'd just have to rebuild b-trees, and that is exactly what our view
> > > > > > >> compactor already does.
> > > > > > >>
> > > > > > >> When we detect a libicu version discrepancy we'd submit the view for
> > > > > > >> compaction. We even have a dedicated "upgrade" [1] channel in smoosh
> > > > > > >> which handles file version format upgrades, but we'll tweak that logic
> > > > > > >> to trigger on libicu version mismatches as well.
> > > > > > >>
> > > > > > >> Would this work? Does anyone see any issue with that approach?
> > > > > > >>
> > > > > > >> [1] https://github.com/apache/couchdb/blob/3.x/src/smoosh/src/smoosh_server.erl#L435-L442
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >> -Nick
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>> On Fri, Oct 29, 2021 at 7:01 PM Nick Vatamaniuc <va...@apache.org> wrote:
> > > > > > >>>
> > > > > > >>> Hello everyone,
> > > > > > >>>
> > > > > > >>> CouchDB by default uses the libicu library to sort its view rows.
> > > > > > >>> When views are built, we do not record or track the version of the
> > > > > > >>> collation algorithm. The issue is that the ICU library may modify the
> > > > > > >>> collation order between major libicu versions, and when that happens,
> > > > > > >>> views built with the older versions may experience data loss. I wanted
> > > > > > >>> to discuss the option to record the libicu collator version in each
> > > > > > >>> view then warn the user when there is a mismatch. Also, optionally
> > > > > > >>> ignore the mismatch, or automatically rebuild the views.
> > > > > > >>>
> > > > > > >>> Imagine, for example, searching patient records using start/end keys.
> > > > > > >>> It could be possible that, say, the first letter of their name now
> > > > > > >>> collates differently in a new libicu. That would prevent the patient
> > > > > > >>> record from showing up in the view results for some important
> > > > > > >>> procedure or medication. Users might not even be aware of this kind of
> > > > > > >>> data loss occurring, there won't be any error in the API or warning in
> > > > > > >>> the logs.
> > > > > > >>>
> > > > > > >>> I was thinking how to solve this. There were a few commits already to
> > > > > > >>> cleanup our collation drivers [1], expose libicu and collation
> > > > > > >>> algorithm version in the new _versions endpoint [2], and some other
> > > > > > >>> minor fixes in that area. As the next steps we could:
> > > > > > >>>
> > > > > > >>> 1) Modify our views to keep track of the collation algorithm
> > > > > > >>> version. We could attempt to transparently upgrade the view header
> > > > > > >>> format -- read the old view file, update the header with an extra
> > > > > > >>> libicu collation version field, that updates the signature, and then,
> > > > > > >>> save the file with the new header and new signature. This avoids view
> > > > > > >>> rebuilds, just records the collator version in the view and moves the
> > > > > > >>> files to a new name.
> > > > > > >>>
> > > > > > >>> 2) Do what PostgreSQL does, and 2a) emit a warning with the view
> > > > > > >>> results when the current libicu version doesn't match the version in
> > > > > > >>> the view [3]. That means altering the view results to add a "warning":
> > > > > > >>> "..." field. Another alternative 2b) is emit a warning in the
> > > > > > >>> _design/$ddoc/_info only. Users would have to know that after an OS
> > > > > > >>> version upgrade, or restoring backups, to make sure to look at their
> > > > > > >>> _design/$ddoc/_info for each db for each ddoc. Of course, there may be
> > > > > > >>> users which used the "raw" collation option, or know they are using
> > > > > > >>> just the plain ASCII character sets in their views. So we'd have a
> > > > > > >>> configuration setting to ignore the warnings as well.
> > > > > > >>>
> > > > > > >>> 3) Users who see the warning, could then either rebuild the view
> > > > > > >>> with the new collator library manually, or it could happen
> > > > > > >>> automatically based on a configuration option, basically "when
> > > > > > >>> collator versions are miss-matched, invalidate and rebuild all the
> > > > > > >>> views".
> > > > > > >>>
> > > > > > >>> 4) We'd have a way for the users to assert (POST a ddoc update) that
> > > > > > >>> they double-checked the new ICU version and are convinced that a
> > > > > > >>> particular view would not experience data loss with the new collator.
> > > > > > >>> That should make the warning go away, and the view to not be rebuilt.
> > > > > > >>> This can't be just a naive "collator" option setting as both per-view
> > > > > > >>> and per-design options are used when computing the view signature, and
> > > > > > >>> any changes there would result in the view being rebuilt. Perhaps we
> > > > > > >>> can add it to the design docs as a separate option which is excluded
> > > > > > >>> from the signature hash, like the "autoupdate" setting for background
> > > > > > >>> index builder ("collation_version_accept"?). PostgreSQL also offers
> > > > > > >>> this option with the ALTER COLLATION ... REFRESH VERSION command [3]
> > > > > > >>>
> > > > > > >>> What do we think, is this a reasonable approach? Is there something
> > > > > > >>> easier / simpler we can do?
> > > > > > >>>
> > > > > > >>> Thanks!
> > > > > > >>> -Nick
> > > > > > >>>
> > > > > > >>> [1] https://github.com/apache/couchdb/pull/3746/commits/28f26f52fe2e170d98658311dafa8198d96b8061
> > > > > > >>> [2] https://github.com/apache/couchdb/commit/c1bb4e4856edd93255d75ebe158b4da38bbf3333
> > > > > > >>> [3] https://www.postgresql.org/docs/13/sql-altercollation.html
> > > > > > >
> > > > > >