You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2020/01/27 11:27:52 UTC
[GitHub] [couchdb] AlexanderKaraberov opened a new issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database file
descriptor process which introduces a performance drop
AlexanderKaraberov opened a new issue #2493: couch_auth_cache creates millions of monitors to _users.couch database file descriptor process which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493
## Description
At one point (11-th of January) we started to get a lot of alerts for `io_queue2.iowait.max` metric, and max `IOWait` time skyrocketed to almost 5 seconds in some cases:
<img width="2202" alt="Screenshot 2020-01-27 at 11 40 41" src="https://user-images.githubusercontent.com/3254818/73168486-35fd0f00-40fa-11ea-9337-4a2f7c7914de.png">
(the drop after 21-th indicates a restart of CouchDB which made the issue to vanish). After some initial investigation in the `remsh` I discovered that top process by reductions was `couch_file` managing `/data/_users.couch` DB which is strange because this seems to be an empty database with only `_design/auth` in there whilst we use a partitioned into 8 shards `_users` database in the `/data/shards/RANGE/_users.timestamp.couch`.
First I decided to log all requests for which [IOWait time here](https://github.com/apache/couchdb-ioq/blob/master/src/ioq_server2.erl#L337) was more than 1 second and all of them were requests going to `_users.couch`:
After profiling `IOQ2` code (we have it enabled + `ioq2.bypass` ) I discovered that delay happens [here for this gen_server:call](https://github.com/apache/couchdb-ioq/blob/master/src/ioq_server2.erl#L109). This suggested me to check monitors of this particular `couch_file` process, and there were 7 million of them! I grouped them by duplicates and I've found out there most of them belonged to `couch_auth_cache`. I do not have a screenshot for that particular case because it was restarted however here's an output from another affected server where there were 2.57 million of these duplicated monitors (`couch_auth_cache`):
<img width="707" alt="monitors" src="https://user-images.githubusercontent.com/3254818/73169736-0996c200-40fd-11ea-931f-ec757d030a45.png">
Quick source code check suggests that the issue might be somewhere in the [couch_auth_cache:reinit_cache](https://github.com/apache/couchdb/blob/2.3.0/src/couch/src/couch_auth_cache.erl) which was called *a lot* of times. When I removed `couch_db:monitor(AuthDb)` and called `gen_server:call(couch_auth_cache, reinit_cache, infinity).` in a loop in the `remsh` for the sake of testing it still created all those duplicated monitors. Hence perhaps the issue lies somewhere in the repeated calls to `open_auth_db`? This suggests that DOWN message was received when `couch_file` for `_users` DB wasn't actually terminated? To verify my hunch I checked the metric for `couchdb.auth_cache_misses` for the same server for the same dates (January 11-th until the restart in the January 21-th) because repeated `reinit`s should result in a cold cache, and here we go:
<img width="2220" alt="cache_misses" src="https://user-images.githubusercontent.com/3254818/73170274-41523980-40fe-11ea-84ae-31e0b16c50d0.png">
As you can see this looks like a perfect mirror for `io_queue2.iowait.max` metric. Any suggestions on how this can be fixed and what can be an initial cause for this `reinit_cache` spike would be great. Also to be honest I don't understand why `couch_auth_cache` still references (uses?) this legacy and empty `_users.couch` DB, because its naming scheme suggests this is something pre-CouchDB 2.0 (not in the shards and no timestamp). Potentially we have some bad misconfiguration somewhere and this flood of `monitor` calls is a bad ramification of it? Thanks!
## Steps to Reproduce
Easy way would be to merely call ``gen_server:call(couch_auth_cache, reinit_cache, infinity).` in a loop until there will be around 7-10 million of duplicated monitors and you start observing slowness. But how this issue happened for us in the wild and on multiple servers I have no clue whatsoever so far. And I suppose this `gen_server:call` slowness can be explained by the fact `monitor`/`demonitor` are not entirely free in the VM due because they involve operations on a red-black tree and doing those on a tree with 7 million nodes can be somewhat slow.
## Expected Behaviour
Only two monitors exist (one for `db_updater` and one for `couch_file`) even in the face of repeated `reinit_cache`.
## Your Environment
* CouchDB version used: 2.3.0
* Operating system and version: Debian 9, Linux kernel 4.9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov commented on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579693850
Apparently this change breaks `cluster_setup` when I start the database from `/dev/run`. It tries to call `POST /_cluster_setup` on port `15984` and crashes here:
```
[error] 2020-01-29T09:25:04.211584Z node1@127.0.0.1 <0.406.0> 29e1339bd9 req_err(4046274191) unknown_error : badarg
[<<"ets:lookup/2">>,<<"couch_auth_cache:exec_if_auth_db/2 L409">>,<<"couch_auth_cache:get_user_creds/2 L63">>,<<"couch_httpd_auth:default_authentication_handler/2 L96">>,<<"chttpd:authenticate_request/2 L531">>,<<"chttpd:process_request/1 L301">>,<<"chttpd:handle_request_int/1 L243">>,<<"mochiweb_http:headers/6 L128">>]
```
From what I see `default_authentication_handler` [still uses couch_auth_cache](https://github.com/apache/couchdb/blob/master/src/couch/src/couch_httpd_auth.erl#L89). I can workaround this but just curious whether this is some sort of a regression or an expected behaviour? The call is sent not to the the `backend_port` (5986) but to a cluster port so I would expect couch_auth_cache to not be used in this particular case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov commented on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579371506
Hello @iilyak ,
Some servers are running `21.3.8.8` and some `21.3.8.11` so that bug shouldn't be related.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579705033
I will close this ticket because the linked PR will prevent my issue from happening again in future, and I'm glad to hear that `couch_auth_cache` will be removed soon. Feel free to reopen if you have some additional insights or piece of advice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579701130
@kocolosk I'm sorry, please never mind my previous comment. This was my mistake 🤦‍♂
I had `{couch_httpd_auth, default_authentication_handler}` set for `chttpd` in my local setup which was redirecting request there. All good now and it works like a charm. We'll upgrade our dev/staging servers to run this cherry-picked change.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] kocolosk commented on issue #2493: couch_auth_cache
creates millions of monitors to _users.couch database which introduces a
performance drop
Posted by GitBox <gi...@apache.org>.
kocolosk commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-578984046
Fascinating. You said that
> When I removed `couch_db:monitor(AuthDb)` line and called `gen_server:call(couch_auth_cache, reinit_cache, infinity).` in a loop in the `remsh` afterwards for the sake of testing it still created all those duplicated monitors.
I can't reproduce that part on master (d59d84ac78e5). If I remove the `couch_db:monitor/1` invocation and call reinit_cache in a loop I only see the one monitor for the file descriptor at any point in time.
If, however, I leave the `couch_db:monitor/1` invocation in place I *do* see monitors piling up against the `couch_db_updater` process when I call reinit in a loop. Is it possible that those monitors were the ones you saw during your debugging, and that the `couch_file` monitors you observed in production have a different root cause?
Taking a step back, it's not at all obvious to me that we still need `couch_auth_cache` as a running service, especially now that the backdoor HTTP port has been removed. I think the `chttpd_auth_cache` module is the one doing the actual user caching. I'd be curious to hear from others on the dev team, but I'm wondering if we can just shut this thing down. Reading through the code it's clear that this module hasn't kept up with the times.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov commented on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579701130
@kocolosk I'm sorry, please never mind my previous comment. This was my mistake 🤦‍♂
I had `{couch_httpd_auth, default_authentication_handler}` set for `chttpd` which was redirecting requests there. All good now and it works like a charm. We'll upgrade our dev/staging servers to run this change.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov commented on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579370645
Hello @kocolosk, and thank you for a fast reply!
> If I remove the couch_db:monitor/1 invocation and call reinit_cache in a loop I only see the one monitor for the file descriptor at any point in time.
Unfortunately this is not a case for me. I'm not running current `master` however I've compared a version of `couch_auth_cache` from our `2.3.0` fork with the current `master` and there are no differences. So here's a normal scenario when I've only launched CouchDB server:
<img width="688" alt="standard" src="https://user-images.githubusercontent.com/3254818/73288725-cd965680-41fb-11ea-802e-c932675dc23a.png">
You might see that there are two monitors currently active for the `couch_auth_cache` process, one is `couch_file` (`<0.320.0>`) and one is `couch_db_updater` (`<0.319.0>`). Now I repeatedly call `reinit_cache` and both of those monitors are duplicated each time:
<img width="393" alt="dupes" src="https://user-images.githubusercontent.com/3254818/73289187-8bb9e000-41fc-11ea-9d73-1602aea1626f.png">
Now when I remove the `couch_db:monitor/1` invocation and repeat the same procedure again I can see that only `couch_file` monitors are duplicated:
<img width="478" alt="couch_file" src="https://user-images.githubusercontent.com/3254818/73289427-fbc86600-41fc-11ea-989b-16b89391d7d7.png">
In fact this matches with what I've seen on our production nodes where there were only `couch_file` monitors duplicates (several million of them) but no duplicated `couch_db_updater` monitors. This suggests that the issue indeed might be related to `open_auth_db` call and not `couch_db:monitor` per se. Potentially those repeated monitors are created by the `couch_db` opener itself? In this case we shouldn't reopen it on `reinit` I suppose?
> Taking a step back, it's not at all obvious to me that we still need couch_auth_cache as a running service, especially now that the backdoor HTTP port has been removed.
This is actually great news. If someone from the dev team can indeed confirm that I can shutdown this process safely without any impact it would be wonderful. I'm no so interested in fixing the root cause of this especially taking into consideration that this is an outdated (legacy) module. I merely don't want this issue to return.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579701130
@kocolosk I'm sorry, please never mind my previous comment. This was my mistake 🤦‍♂
I had `{couch_httpd_auth, default_authentication_handler}` set for `chttpd` in my local setup which was redirecting request there. All is good now and it works like a charm. We'll upgrade our dev/staging servers to run this cherry-picked change and if there are no regressions, production later.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov commented on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579705033
I will close this ticket because the linked PR solves my issue and I'm glad to hear that `couch_auth_cache` will be removed soon. Feel free to reopen if you have some additional insights or piece of advice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov closed issue #2493: couch_auth_cache
creates millions of monitors to _users.couch database which introduces a
performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov closed issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] iilyak commented on issue #2493: couch_auth_cache creates
millions of monitors to _users.couch database which introduces a
performance drop
Posted by GitBox <gi...@apache.org>.
iilyak commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579214468
Which version of erlang you are running. The reason I ask because there was a major issue with behavior of monitor/demonitor feature in 20.3.8.5 ([see ERL-751](https://bugs.erlang.org/browse/ERL-751))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579656736
Thanks! This looks good to me. I'll give it a try
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579370645
Hello @kocolosk, and thank you for a fast reply!
> If I remove the couch_db:monitor/1 invocation and call reinit_cache in a loop I only see the one monitor for the file descriptor at any point in time.
Unfortunately this is not a case for me. I'm not running current `master` however I've compared a version of `couch_auth_cache` from our `2.3.0` fork with the current `master` and there are no differences. So here's a normal scenario when I've only launched CouchDB server:
<img width="688" alt="standard" src="https://user-images.githubusercontent.com/3254818/73288725-cd965680-41fb-11ea-802e-c932675dc23a.png">
You might see that there are two monitors currently active for the `couch_auth_cache` process, one is `couch_file` (`<0.320.0>`) and one is `couch_db_updater` (`<0.319.0>`). Now I repeatedly call `reinit_cache` and both of those monitors are duplicated each time:
<img width="393" alt="dupes" src="https://user-images.githubusercontent.com/3254818/73289187-8bb9e000-41fc-11ea-9d73-1602aea1626f.png">
Now when I remove the `couch_db:monitor/1` invocation and repeat the same procedure again I can see that only `couch_file` monitors are duplicated:
<img width="478" alt="couch_file" src="https://user-images.githubusercontent.com/3254818/73289427-fbc86600-41fc-11ea-989b-16b89391d7d7.png">
In fact this matches with what I've seen on our production nodes where there were only `couch_file` monitors duplicates (several million of them) but no duplicated `couch_db_updater` monitors.
> Is it possible that those monitors were the ones you saw during your debugging, and that the couch_file monitors you observed in production have a different root cause?
I'm not fully excluding this possibility. However one interesting correlation as per my first post is that accumulating of these monitors and subsequent slow `pread_iolist` calls exactly correspond to a spike in `couchdb.auth_cache_misses`. What else could trigger this if not a cold cache?
Potentially I'm missing some changes between `master` and 2.3.0 which addressed this issue?
> Taking a step back, it's not at all obvious to me that we still need couch_auth_cache as a running service, especially now that the backdoor HTTP port has been removed.
This is actually great news. If someone from the dev team can indeed confirm that I can shutdown this process safely without any impact it would be wonderful. I'm no so interested in fixing the root cause of this especially taking into consideration that this is an outdated (legacy) module. I merely don't want this issue to return.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579370645
Hello @kocolosk, and thank you for a fast reply!
> If I remove the couch_db:monitor/1 invocation and call reinit_cache in a loop I only see the one monitor for the file descriptor at any point in time.
Unfortunately this is not a case for me. I'm not running current `master` however I've compared a version of `couch_auth_cache` from our `2.3.0` fork with the current `master` and there are no differences. So here's a normal scenario when I've only launched CouchDB server:
<img width="688" alt="standard" src="https://user-images.githubusercontent.com/3254818/73288725-cd965680-41fb-11ea-802e-c932675dc23a.png">
You might see that there are two monitors currently active for the `couch_auth_cache` process, one is `couch_file` (`<0.320.0>`) and one is `couch_db_updater` (`<0.319.0>`). Now I repeatedly call `reinit_cache` and both of those monitors are duplicated each time:
<img width="393" alt="dupes" src="https://user-images.githubusercontent.com/3254818/73289187-8bb9e000-41fc-11ea-9d73-1602aea1626f.png">
Now when I remove the `couch_db:monitor/1` invocation and repeat the same procedure again I can see that only `couch_file` monitors are duplicated:
<img width="478" alt="couch_file" src="https://user-images.githubusercontent.com/3254818/73289427-fbc86600-41fc-11ea-989b-16b89391d7d7.png">
In fact this matches with what I've seen on our production nodes where there were only `couch_file` monitors duplicates (several million of them) but no duplicated `couch_db_updater` monitors.
> Is it possible that those monitors were the ones you saw during your debugging, and that the couch_file monitors you observed in production have a different root cause?
I'm not fully excluding this possibility. However one interesting correlation as per my first post is that accumulating of these monitors and subsequent slow `pread_iolist` calls exactly correspond to a spike in `couchdb.auth_cache_misses`. What else could trigger this if not a cold cache?
> Taking a step back, it's not at all obvious to me that we still need couch_auth_cache as a running service, especially now that the backdoor HTTP port has been removed.
This is actually great news. If someone from the dev team can indeed confirm that I can shutdown this process safely without any impact it would be wonderful. I'm no so interested in fixing the root cause of this especially taking into consideration that this is an outdated (legacy) module. I merely don't want this issue to return.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579693850
Apparently this change breaks `cluster_setup` when I start the database from `/dev/run`. It tries to call `POST /_cluster_setup` on port `15984` and crashes here:
```
[error] 2020-01-29T09:25:04.211584Z node1@127.0.0.1 <0.406.0> 29e1339bd9 req_err(4046274191) unknown_error : badarg
[<<"ets:lookup/2">>,<<"couch_auth_cache:exec_if_auth_db/2 L409">>,<<"couch_auth_cache:get_user_creds/2 L63">>,<<"couch_httpd_auth:default_authentication_handler/2 L96">>,<<"chttpd:authenticate_request/2 L531">>,<<"chttpd:process_request/1 L301">>,<<"chttpd:handle_request_int/1 L243">>,<<"mochiweb_http:headers/6 L128">>]
```
From what I see `default_authentication_handler` [still uses couch_auth_cache](https://github.com/apache/couchdb/blob/master/src/couch/src/couch_httpd_auth.erl#L89). I can workaround this but just curious whether this is some sort of a regression or an expected behaviour? `/cluster_setup` request is sent not to the `backend_port` (5986) but to a cluster port so I would expect `couch_auth_cache` to not be used in this particular case.
*UPD:* This my fault. I had a wrong `local.ini` config. All works good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579701130
@kocolosk I'm sorry, please never mind my previous comment. This was my mistake 🤦‍♂
I had `{couch_httpd_auth, default_authentication_handler}` set for `chttpd` which was redirecting requests there. All good now and it works like a charm. We'll upgrade our dev/staging servers to run this cherry-picked change.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] wohali commented on issue #2493: couch_auth_cache creates
millions of monitors to _users.couch database which introduces a
performance drop
Posted by GitBox <gi...@apache.org>.
wohali commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-578994076
@kocolosk, @rnewson would know for sure when it comes to the removal of 5986 and this, but is the auth all handled by chttpd or is there a secondary check once it hits what's left of couch_httpd? if the former I concur with you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] kocolosk commented on issue #2493: couch_auth_cache
creates millions of monitors to _users.couch database which introduces a
performance drop
Posted by GitBox <gi...@apache.org>.
kocolosk commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579505404
Reviewing the code it seemed to me that `couch_auth_cache` was safe to remove as a running service, so that’s what I did in #2498. I had to disable a few tests that were written to run against the “backdoor” port, but didn’t see any issues with the rest of the test suite. If you feel like cherry-picking f4356a32d37e0c26b1423026bd17fc0c94c6dcb5 I suspect you can do so safely
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov commented on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov commented on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579656736
Thanks! This looks good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579693850
Apparently this change breaks `cluster_setup` when I start the database from `/dev/run`. It tries to call `POST /_cluster_setup` on port `15984` and crashes here:
```
[error] 2020-01-29T09:25:04.211584Z node1@127.0.0.1 <0.406.0> 29e1339bd9 req_err(4046274191) unknown_error : badarg
[<<"ets:lookup/2">>,<<"couch_auth_cache:exec_if_auth_db/2 L409">>,<<"couch_auth_cache:get_user_creds/2 L63">>,<<"couch_httpd_auth:default_authentication_handler/2 L96">>,<<"chttpd:authenticate_request/2 L531">>,<<"chttpd:process_request/1 L301">>,<<"chttpd:handle_request_int/1 L243">>,<<"mochiweb_http:headers/6 L128">>]
```
From what I see `default_authentication_handler` uses [couch_auth_cache here](https://github.com/apache/couchdb/blob/master/src/couch/src/couch_httpd_auth.erl#L89). I can workaround this but just curious whether this is some sort of a regression or an expected behaviour? `/cluster_setup` request is sent not to the `backend_port` (5986) but to a cluster port so I would expect `couch_auth_cache` to not be used in this particular case.
*UPD:* This my fault. I had a wrong `local.ini` config. All works good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] AlexanderKaraberov edited a comment on issue #2493:
couch_auth_cache creates millions of monitors to _users.couch database
which introduces a performance drop
Posted by GitBox <gi...@apache.org>.
AlexanderKaraberov edited a comment on issue #2493: couch_auth_cache creates millions of monitors to _users.couch database which introduces a performance drop
URL: https://github.com/apache/couchdb/issues/2493#issuecomment-579693850
Apparently this change breaks `cluster_setup` when I start the database from `/dev/run`. It tries to call `POST /_cluster_setup` on port `15984` and crashes here:
```
[error] 2020-01-29T09:25:04.211584Z node1@127.0.0.1 <0.406.0> 29e1339bd9 req_err(4046274191) unknown_error : badarg
[<<"ets:lookup/2">>,<<"couch_auth_cache:exec_if_auth_db/2 L409">>,<<"couch_auth_cache:get_user_creds/2 L63">>,<<"couch_httpd_auth:default_authentication_handler/2 L96">>,<<"chttpd:authenticate_request/2 L531">>,<<"chttpd:process_request/1 L301">>,<<"chttpd:handle_request_int/1 L243">>,<<"mochiweb_http:headers/6 L128">>]
```
From what I see `default_authentication_handler` [still uses couch_auth_cache](https://github.com/apache/couchdb/blob/master/src/couch/src/couch_httpd_auth.erl#L89). I can workaround this but just curious whether this is some sort of a regression or an expected behaviour? `/cluster_setup` request is sent not to the `backend_port` (5986) but to a cluster port so I would expect `couch_auth_cache` to not be used in this particular case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services