You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Miles Pomeroy <po...@gmail.com> on 2011/12/06 19:03:41 UTC
User authentication failures
I'm having this weird issue where user authentication fails. It has
happened twice so far on one of our development couchdb servers. As an
admin I can still log in, but for normal users their authentication
fails. If I change the password, authentication still fails. The only
thing I've found that works, is restarting the server. Then
authentication works just fine again. Any ideas on what's causing it
or where to look for the problem. I haven't seen anything obvious in
the log file.
Miles.
Re: User authentication failures
Posted by Pete Vander Giessen <pe...@gmail.com>.
On Wed, Dec 7, 2011 at 1:28 PM, CGS <cg...@gmail.com> wrote:
> Ah! I forgot to answer your question related to replication, but I think the
> information I provided to you from my test answers also that question
> indirectly.
It did (indirectly). Thank you very much for all the info. I will
test, and post back to the list with results :-)
~ PeteVG
Re: User authentication failures
Posted by CGS <cg...@gmail.com>.
Ah! I forgot to answer your question related to replication, but I think
the information I provided to you from my test answers also that
question indirectly.
CGS
On 12/07/2011 10:10 PM, CGS wrote:
> I tested on 2 GB RAM increasing every time by one order of magnitude
> at the time. At 10M it no longer started, complaining about not being
> able to allocate resources. I don't know exactly if it is the RAM or
> my linux file descriptors (CentOS 6.0). So, I remained at 1M. As far
> as I know, Windows 7 supports at maximum 16M file descriptors, so, you
> may try to raise it up to that limit for the test.
>
> Another test which I've done was if I can really have 1M spawned
> threads which can interact with the OS (Erlang process in which I
> started 1M processes in parallel which each should have brought
> something with cURL). The answer was no (the OS stopped increasing the
> number of cURL instances, invoking lack of resources). The full memory
> was occupied and so I couldn't determine if I reached the OS limit or
> I was starving my processes.
>
> In case the RAM is the problem, I suppose your 8 GB can bring up a lot
> of threads, but if the OS is imposing the limit, then you may want to
> leave some file descriptors for other processes as well.
>
> I would be interested at what numbers you reach. :)
>
> CGS
>
>
>
> On 12/07/2011 08:39 PM, Pete Vander Giessen wrote:
>> On Wed, Dec 7, 2011 at 12:38 AM, CGS<cg...@gmail.com> wrote:
>>> It looks like Erlang reached the 1024 processes default limit. That
>>> should
>>> be solved by increasing ERL_MAX_PORTS ... To be noted that
>>> Erlang checks for the working memory availability for each thread,
>>> so, the
>>> maximum ports cannot be insanely high.
>> Thank you for that piece of info. How high would you say would be
>> "insanely" high for, say, a server w/ 8GB of RAM?
>>
>> Broader question: if I start a continuous replication, will that hold
>> a thread open for the (theoretically unlimited) duration of the
>> replication?
>>
>> They key thing is that I'm trying to figure out how we hit that limit
>> without doing anything _too_ mean to the server. Increasing the thread
>> limit sounds like it might be part of the solution, but I'd like to
>> understand the problem a bit more, and I'm doing a bit of blind
>> flailing at the moment ...
>>
>> ~ PeteVG
>
Re: User authentication failures
Posted by CGS <cg...@gmail.com>.
I tested on 2 GB RAM increasing every time by one order of magnitude at
the time. At 10M it no longer started, complaining about not being able
to allocate resources. I don't know exactly if it is the RAM or my linux
file descriptors (CentOS 6.0). So, I remained at 1M. As far as I know,
Windows 7 supports at maximum 16M file descriptors, so, you may try to
raise it up to that limit for the test.
Another test which I've done was if I can really have 1M spawned threads
which can interact with the OS (Erlang process in which I started 1M
processes in parallel which each should have brought something with
cURL). The answer was no (the OS stopped increasing the number of cURL
instances, invoking lack of resources). The full memory was occupied and
so I couldn't determine if I reached the OS limit or I was starving my
processes.
In case the RAM is the problem, I suppose your 8 GB can bring up a lot
of threads, but if the OS is imposing the limit, then you may want to
leave some file descriptors for other processes as well.
I would be interested at what numbers you reach. :)
CGS
On 12/07/2011 08:39 PM, Pete Vander Giessen wrote:
> On Wed, Dec 7, 2011 at 12:38 AM, CGS<cg...@gmail.com> wrote:
>> It looks like Erlang reached the 1024 processes default limit. That should
>> be solved by increasing ERL_MAX_PORTS ... To be noted that
>> Erlang checks for the working memory availability for each thread, so, the
>> maximum ports cannot be insanely high.
> Thank you for that piece of info. How high would you say would be
> "insanely" high for, say, a server w/ 8GB of RAM?
>
> Broader question: if I start a continuous replication, will that hold
> a thread open for the (theoretically unlimited) duration of the
> replication?
>
> They key thing is that I'm trying to figure out how we hit that limit
> without doing anything _too_ mean to the server. Increasing the thread
> limit sounds like it might be part of the solution, but I'd like to
> understand the problem a bit more, and I'm doing a bit of blind
> flailing at the moment ...
>
> ~ PeteVG
Re: User authentication failures
Posted by Pete Vander Giessen <pe...@gmail.com>.
On Wed, Dec 7, 2011 at 12:38 AM, CGS <cg...@gmail.com> wrote:
>
> It looks like Erlang reached the 1024 processes default limit. That should
> be solved by increasing ERL_MAX_PORTS ... To be noted that
> Erlang checks for the working memory availability for each thread, so, the
> maximum ports cannot be insanely high.
Thank you for that piece of info. How high would you say would be
"insanely" high for, say, a server w/ 8GB of RAM?
Broader question: if I start a continuous replication, will that hold
a thread open for the (theoretically unlimited) duration of the
replication?
They key thing is that I'm trying to figure out how we hit that limit
without doing anything _too_ mean to the server. Increasing the thread
limit sounds like it might be part of the solution, but I'd like to
understand the problem a bit more, and I'm doing a bit of blind
flailing at the moment ...
~ PeteVG
Re: User authentication failures
Posted by CGS <cg...@gmail.com>.
It looks like Erlang reached the 1024 processes default limit. That
should be solved by increasing ERL_MAX_PORTS (and, if it doesn't help,
add also the parameter +P number_of_max_ports to erl command). I had
similar problems with another Erlang software and that solved the
problem. To be noted that Erlang checks for the working memory
availability for each thread, so, the maximum ports cannot be insanely high.
CGS
On 12/06/2011 11:49 PM, Pete Vander Giessen wrote:
> Hi All,
>
>> I created an issue to track your problem:
>>
>> https://issues.apache.org/jira/browse/COUCHDB-1357
>>
>> Is there a deterministic way to reproduce your issue?
>> A sequence of exact steps, or a script, would be the ideal.
> We ran into a similar issue, with 1.1.1. The problem seemed to start
> with a log entry that looked like so:
>
> [Thu, 01 Dec 2011 19:22:48 GMT] [error] [<0.31233.1889>] Uncaught
> error in HTTP request: {error,system_limit}
> [Thu, 01 Dec 2011 19:22:48 GMT] [info] [<0.31233.1889>] Stacktrace:
> [{erlang,open_port,
> [{spawn,"couch_icu_driver"},[]]},
> {couch_util,drv_port,0},
> {couch_util,collate,3},
> {couch_view,less_json_ids,2},
> {couch_btree,drop_nodes,4},
> {couch_btree,stream_kp_node,8},
> {couch_btree,fold,4},
> {couch_view,fold,4}]
>
> We had a cluster of 7 or so Windows users replicating 2 databases
> apiece from a "master" server running on Red Hat. Each user also has a
> separate "user" database that they use to fetch oauth credentials, so
> there were perhaps 16 databases in use at the time.
>
> Next time this happens, I'll run netstat on the machine and save the
> output. For now, I've saved off the logs, and can provide more
> complete logs upon request.
>
> I'll cross-post this as a comment to the bug (and I'm logged into the
> #couchdb channel as "petevg" if anybody has realtime questions for
> me).
>
> ~ PeteVG
Re: User authentication failures
Posted by Pete Vander Giessen <pe...@gmail.com>.
Hi All,
> I created an issue to track your problem:
>
> https://issues.apache.org/jira/browse/COUCHDB-1357
>
> Is there a deterministic way to reproduce your issue?
> A sequence of exact steps, or a script, would be the ideal.
We ran into a similar issue, with 1.1.1. The problem seemed to start
with a log entry that looked like so:
[Thu, 01 Dec 2011 19:22:48 GMT] [error] [<0.31233.1889>] Uncaught
error in HTTP request: {error,system_limit}
[Thu, 01 Dec 2011 19:22:48 GMT] [info] [<0.31233.1889>] Stacktrace:
[{erlang,open_port,
[{spawn,"couch_icu_driver"},[]]},
{couch_util,drv_port,0},
{couch_util,collate,3},
{couch_view,less_json_ids,2},
{couch_btree,drop_nodes,4},
{couch_btree,stream_kp_node,8},
{couch_btree,fold,4},
{couch_view,fold,4}]
We had a cluster of 7 or so Windows users replicating 2 databases
apiece from a "master" server running on Red Hat. Each user also has a
separate "user" database that they use to fetch oauth credentials, so
there were perhaps 16 databases in use at the time.
Next time this happens, I'll run netstat on the machine and save the
output. For now, I've saved off the logs, and can provide more
complete logs upon request.
I'll cross-post this as a comment to the bug (and I'm logged into the
#couchdb channel as "petevg" if anybody has realtime questions for
me).
~ PeteVG
Re: User authentication failures
Posted by Filipe David Manana <fd...@apache.org>.
Hi Miles,
I created an issue to track your problem:
https://issues.apache.org/jira/browse/COUCHDB-1357
Is there a deterministic way to reproduce your issue?
A sequence of exact steps, or a script, would be the ideal.
Before that happened, did you have any stack traces in the log?
On Tue, Dec 6, 2011 at 6:04 PM, Miles Pomeroy <po...@gmail.com> wrote:
> Using version 1.1.1
--
Filipe David Manana,
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: User authentication failures
Posted by Miles Pomeroy <po...@gmail.com>.
Using version 1.1.1