You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "sergey-safarov (via GitHub)" <gi...@apache.org> on 2024/02/26 22:44:31 UTC

[I] view has the wrong signature [couchdb]

sergey-safarov opened a new issue, #4994:
URL: https://github.com/apache/couchdb/issues/4994

   ## Description
   
   On one node (db-2.example.com) we catch error logs
   > [error] 2024-02-26T19:53:42.735715Z couchdb@db-2.example.com <0.21142.1038> -------- ./data/.shards/e0000000-ffffffff/account/a1/15/78067a1db9f83fa4da53f6bacc16.1663539188_design/mrview/fa7a1a9e5db7f6873529299ea929daad.view has the wrong signature: expected: <<250,122,26,158,93,183,246,135,53,41,41,158,169,41,218,173>> but got <<87,225,55,84,101,143,50,221,45,61,42,165,141,102,202,177>>
   [error] 2024-02-26T19:54:07.738902Z couchdb@db-2.example.com <0.22173.1076> -------- ./data/.shards/40000000-5fffffff/account/a1/15/78067a1db9f83fa4da53f6bacc16.1663539188_design/mrview/fa7a1a9e5db7f6873529299ea929daad.view has the wrong signature: expected: <<250,122,26,158,93,183,246,135,53,41,41,158,169,41,218,173>> but got <<87,225,55,84,101,143,50,221,45,61,42,165,141,102,202,177>>
   
   Then it triggers CPU usage and is not responsible CouchDB 3 nodes cluster.
   
   In logs present messages like
   fabric_worker_timeout get_db_info
   > [error] 2024-02-26T20:00:39.603653Z couchdb@db-2.example.com <0.26790.1086> 182b91a95a fabric_worker_timeout get_db_info,'couchdb@db-0.example.com',<<"shards/20000000-3fffffff/account/56/3b/1e36302175513343e6a13f6a1372-202308.1690848013">>
   
   fabric_worker_timeout open_doc
   > [error] 2024-02-26T20:00:40.479851Z couchdb@db-2.example.com <0.2074.1115> -------- fabric_worker_timeout open_doc,'couchdb@db-2.example.com',<<"shards/40000000-5fffffff/account/e3/2d/d743a02e6ee4e28f02755e070d33.1685539862">>
   
   fabric_worker_timeout open_doc
   > [error] 2024-02-26T20:00:45.480715Z couchdb@db-2.example.com <0.26670.936> a6eca4ce74 fabric_worker_timeout open_doc,'couchdb@db-1.example.com',<<"shards/40000000-5fffffff/account/e3/2d/d743a02e6ee4e28f02755e070d33.1685539862">>
   
   ## Steps to Reproduce
   
   Not known.
   
   ## Expected Behaviour
   
   Error in the one view should not stop functionality of 3 nodes cluster.
   
   ## Your Environment
   
   * CouchDB version used: 3.3.2
   * Browser name and version: browser do not used
   * Operating system and version: CentOS 8
   
   ## Additional Context
   
   Used `apache/couchdb:3.3.2` docker container.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2260369446

   jan's theory makes sense to me. the 2.x -> 3.x upgrade code is flawed, it only tries the 2.0 sig if the file was found in the old location, and assumes we'll write a new header for all views on upgrade (any couchdb crash or restart would cause things to go wrong)
   
   suggest a different approach at https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L121 or so, we can try to match the v2 sig if the v3 sig fails. that is, decouple the moving of the file from old to new location from whether the file itself is v2 vs v3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-1966458894

   > after the view rebuilds, it should be back to normal.
   
   Also, an issue with the `view` file happens on one node in the cluster. But this does not explain why CPU consumed on other nodes.
   So view rebuild should happen on one node and two other nodes should be not affected.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2040160442

   I searched through our logs and found a few of "has the wrong signature" errors as well. In our case they all happened on nodes were being decommissioned and database shards were migrating to new nodes. Wonder if there is a higher chance of it happening if there is a network partition or shard map changes when a the design document updates at the same time...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034864899

   fixed since 3.3.3 with commit https://github.com/apache/couchdb/commit/453c698a51b36b68e316a7f772d9120bd492670f


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034898631

   @nickva term_to_binary is fragile and we should not be using it for view signatures (at least). The original decision was done I'm sure out of expediency and/or (benign) ignorance. forcing {minor_version, 1} is likely a solid fix for years to come, but the only truly safe path out is to define the view signature algorithm explicitly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034589442

   We have reproduced the same issue on the server without docker.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-1997609511

   I am not sure exactly why more CPU would be consumed on the other nodes. It could be that view shards there hadn't caught up as much as the shard which was reset so they started to get built. When this view was reset perhaps there were active requests waiting to receive rows (responses) from it and so waiting view clients might have been piling up.
   
   You can check for the number of waiting view clients with: https://docs.couchdb.org/en/stable/api/ddoc/common.html#get--db-_design-ddoc-_info
   
   To see if it's any view builds taking place, try using https://docs.couchdb.org/en/stable/api/server/common.html#active-tasks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "jcoglan (via GitHub)" <gi...@apache.org>.
jcoglan commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034724358

   It appears that it is possible for the design doc signature to vary according to something we have not yet determined. The client's cluster that we (Neighbourhoodie) are investigating gives the following results for one of the design docs affected by a bad index signature:
   
   $ cdb '/' | jq '{ version, git_sha }'
   {
     "version": "3.3.3",
     "git_sha": "40afbcfc7"
   }
   
   $ cdb '/{db}/_design/{id}' | md5sum
   ef932f29b40bc1795360ea095051d782  -
   
   $ cdb '/{db}/_design/{id}/_info' | jq '.view_index.signature'
   "c15a2c0300aefa07abdea125eba80a98"
   
   We copied this design doc and saved into a local dev cluster running the same CouchDB version/commit, and it gave the same md5sum and signature.
   
   However, when putting this same design doc into the CouchDB service installed by Homebrew, it gives the same md5sum but a different signature:
   
   $ cdb '/{db}/_design/{id}/_info' | jq '.view_index.signature'
   "20ae103a2d660c16f4cbbc43703ede09"
   
   This indicates that it’s possible for the same design document to produce a different signature on different systems, possibly on different nodes of the same cluster depending on what the cause is.
   
   Environment of each test:
   
   - Customer production system: Linux, CouchDB v3.3.3, SpiderMonkey 1.8.5, Erlang 24.3.4.15, ICU library 60.3, collator 153.80, algorithm 10
   
   - Local dev cluster: macOS 13.6.4, CouchDB v3.3.3, SpiderMonkey 91, Erlang 25.3.2.9, ICU library 74.2, collator 153.121, algorithm 15.1
   
   - Homebrew install: macOS 13.6.4, CouchDB v3.3.3, SpiderMonkey 91, Erlang 26.2.1, ICU library 73.2, collator 153.120, algorithm 15


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2037289935

   For me, it isn't easy to reproduce. It randomly happens.
   The CouchDB server works several weeks without this error and then it can happen.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "janl (via GitHub)" <gi...@apache.org>.
janl commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2045270650

   > noting that couchdb 3.3.3 explicitly rejects OTP 26;
   
   I seem to have been naughty when making the 3.3.3 Mac binaries, I swear I did this for a good reason, but I don’t recall it at the moment. I think the 25-jit failed on arm Mac, but I’m not sure: https://github.com/janl/build-couchdb-mac/blob/master/build.sh#L72-L73
   
   None of this is relevant to the issue in the ticket, it just clarifies what @jcoglan reported


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034889690

   Good find @rnewson it's the `{minor_version, 1}` and atom encoding. Any terms with atoms produced running with Erlang 26 before that would definitely have the a different hash.
   
   On 26
   ```
   > term_to_binary(foo, [{minor_version,1}]).
   <<131,100,0,3,102,111,111>>
   
   > term_to_binary(foo).
   <<131,119,3,102,111,111>>
   ```
   
   But I wonder if there is something else besides atoms there that would cause in-determinism, I'd hope not because that would be terrible. We don't use maps there, ref and pids but binary reference chunks might sneak in...?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034869105

   ```
   rebar.config.script:    {require_otp_vsn, "23|24|25"},
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034862858

   ```
   (node1@127.0.0.1)17> Views.
   [{mrview,0,0,0,[],
            [{<<"consumed">>,<<"_sum">>}],
            <<"function (doc) { if (doc.pvt_type != 'allotment_consumption' || doc.pvt_deleted) ret"...>>,
            nil,[]},
    {mrview,1,0,0,
            [<<"consumed_by_callid">>],
            [],
            <<"function (doc) { if (doc.pvt_type != 'allotment_consumption' || doc.pvt_deleted)"...>>,
            nil,[]}]
   (node1@127.0.0.1)18> Language.
   <<"javascript">>
   (node1@127.0.0.1)19> DesignOpts.
   []
   (node1@127.0.0.1)20> Libs.
   []
   (node1@127.0.0.1)21> couch_util:to_hex_bin(couch_hash:md5_hash(term_to_binary({Views, Language, DesignOpts, Libs}))).
   <<"c15a2c0300aefa07abdea125eba80a98">>
   (node1@127.0.0.1)22> couch_util:to_hex_bin(couch_hash:md5_hash(term_to_binary({Views, Language, DesignOpts, Libs}, [{minor_version, 2}]))).
   <<"20ae103a2d660c16f4cbbc43703ede09">>
   ```
   
   couchdb 3.3.3 does not use term_to_bin everywhere, so its output will vary based on whether it runs inside OTP 26 or earler.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-1966355502

   Just checked via AWS console, the CouchDB has "Multi-Attach enabled: no".
   So  use volume mounted to only one node.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2036064986

   > We have reproduced the same issue on the server without docker.
   
   @sergey-safarov that's good news. Can it turned into a script or it's fairly simple to describe steps?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034867697

   noting that couchdb 3.3.3 explicitly rejects OTP 26;
   
   ```
   make
   ==> snappy (compile)
   ERROR: OTP release 26 does not match required regex 23|24|25
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-1965495755

   I would also ask that question: have the files been moved or renamed outside of couchdb's control? This error is not normal, and is a protection mechanism (effectively an assertion). another possibility is you have multiple nodes pointing at the same shared volume, and trashing each others state.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "janl (via GitHub)" <gi...@apache.org>.
janl commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2048319508

   @jcoglan and I have come up with a repro for the 'wrong signature' event: https://gist.github.com/jcoglan/0a5feb4af2a496ce10c9b80cf02ea28f. Our theory is that when an index is first queried following a v2->v3 upgrade, https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L121 will rename the file and return the old signature. however, maybe_update_index_file/1 just renames the file, it does not change its content. https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L127 matches and the normal index update path is followed, but if the index is empty then no new content and no new header is written, so the old signature remains in the file. the next time the view is queried, maybe_update_index_file/1 will do nothing (the old file does not exist) and return ok, so we hit this clause where wrong signature appears https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L139


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2034858367

   @jcoglan I suspect if term_to_binary output is different between architectures or erlang versions somehow.
   
   View signatures are computed in: https://github.com/apache/couchdb/blob/33bfa1328ed7f20de2983c9394fb473cf4ca5ecf/src/couch_mrview/src/couch_mrview_util.erl#L280-L281
   
   See if you can an add a log statement and dump the `SigInfo` term there and try running the term through the term_to_binary in an erlang prompt on all erlang/architecture versions and see it  produces the same result.
   
   Erlang 26 had introduced the `deterministic` term_to_binary option so maybe we should start using that (though it only guarantees to be stable for the same Erlang/OTP release).
   
   The other thing that's changed is UTF8 encoding of atoms. That's probably what the problem is here.
   
   See some discussion from last year about it: https://github.com/apache/couchdb/issues/4467#issuecomment-1464005748
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-1966342416

   >  Is there any chance your view shard files were moved, copied, restored from backup from an much older couch instance, or mounted on a volume shared across multiple nodes?
   
   I do not think we make something from the above.
   We starting docker container and do not move CouchDB files.
   Also, we do not use shared volumes.
   We make couchdb backup using `bulk` requests but do not restore CouchdDB database at this time period.
   
   > Is it easy to reproduce?  Just happened once or it's a regular occurrence?
   It is the first time.
   
   > after the view rebuilds, it should be back to normal.
   
   This does not happen for 17 minutes and we have recreated the docker container.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-1965492535

   That happens when a view shard is opened but has a view signature that's not current. It comes from https://github.com/apache/couchdb/blob/29db2df901207d67e1975ca11a091bcb7f46abcc/src/couch_mrview/src/couch_mrview_index.erl#L141
   
   The expected signature matches file path:
   
   ```
   binary:encode_hex(<<250,122,26,158,93,183,246,135,53,41,41,158,169,41,218,173>>).
   <<"FA7A1A9E5DB7F6873529299EA929DAAD">>
   ```
   
   The other one is:
   ```
   binary:encode_hex(<<87,225,55,84,101,143,50,221,45,61,42,165,141,102,202,177>>).
   <<"57E13754658F32DD2D3D2AA58D66CAB1">>
   ```
   
   I don't recall seeing this error too often. Is there any chance your view shard files were moved, copied, restored from backup from an much older couch instance, or mounted on a volume shared across multiple nodes?
   
   Is it easy to reproduce? Just happened once or it's a regular occurrence?
   
   > Then it triggers CPU usage and is not responsible CouchDB 3 nodes cluster.
   
   That's expected as the next action after the log is to reset the view shard and rebuild. So in other words, after the view rebuilds, it should be back to normal.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] view has the wrong signature [couchdb]

Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #4994:
URL: https://github.com/apache/couchdb/issues/4994#issuecomment-2260137918

   It just reproduced the issue.
   The issue happened during the CouchDB upgrade from 2.3.1 to 3.3.1.
   I have started daemon. Daemon failed because the user permission for the data folder was wrong. While updating the the CouchDB daemon was stopped on the old server, detached volume from the old server, attached to the new server, and started CouchDB on the new server. On user changed `uid` for `couchdb` user and daemons failed because does not have permission to `data` folder content.
   I was making `chown` for the `data` folder content and then `couchdb` service started properly and the issue reproduced.
   Now I can see in the logs
   ```
   couchdb@db-0.example.com <0.3795.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/462e71ffa4b5d3b38b811d08ad72396c.view has the wrong signature: expected: <<70,46,113,255,164,181,211,179,139,129,29,8,173,114,57,108>> but got <<2,65,77,140,1,52,42,68,196,86,58,98,97,23,145,106>>
   couchdb@db-0.example.com <0.3922.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/dd51edd8663a1cc90af2f8215cd797be.view has the wrong signature: expected: <<221,81,237,216,102,58,28,201,10,242,248,33,92,215,151,190>> but got <<174,153,108,105,115,213,43,72,131,221,10,174,163,141,228,115>>
   couchdb@db-0.example.com <0.3934.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/27fcce55829ac4f09ab0f195d43ad529.view has the wrong signature: expected: <<39,252,206,85,130,154,196,240,154,176,241,149,212,58,213,41>> but got <<241,187,13,21,148,238,32,71,82,114,244,157,164,178,210,214>>
   couchdb@db-0.example.com <0.4095.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/b9a49bb8b24a083e07c436f17397fb54.view has the wrong signature: expected: <<185,164,155,184,178,74,8,62,7,196,54,241,115,151,251,84>> but got <<255,60,196,213,171,156,94,131,41,80,206,111,60,138,78,212>>
   couchdb@db-0.example.com <0.2473.4> -------- ./data/.shards/c0000000-dfffffff/port_requests.1595155200_design/mrview/d2d3acac282c8c10e0edc3102682ae07.view has the wrong signature: expected: <<210,211,172,172,40,44,140,16,224,237,195,16,38,130,174,7>> but got <<91,44,84,255,172,124,87,248,145,234,222,14,180,50,2,102>>
   couchdb@db-0.example.com <0.5771.4> -------- ./data/.shards/00000000-1fffffff/port_requests.1595155200_design/mrview/d2d3acac282c8c10e0edc3102682ae07.view has the wrong signature: expected: <<210,211,172,172,40,44,140,16,224,237,195,16,38,130,174,7>> but got <<91,44,84,255,172,124,87,248,145,234,222,14,180,50,2,102>>
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org