You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Gregor Martynus <gr...@martynus.net> on 2012/01/05 10:18:49 UTC
couchDB with databases per user: continuous replications stop
working, without errors
I experience massive reliability issues with continuous replications in my
app. Each user has an own profile db with a continuous replication to a
central database. Numbers: ~1800 dbs & ~2600 continuous replications (using
_replicator db), ~200 did not get started.
My hope is that others faced these issues before and found ways to fix or
work around them. So if you are a developer or know developers that have a
similar couchDB setup, would you mind to get me in touch? Thanks for your
help!
Tech info:
couchDB 1.1.1 (iriscouch)
max_dbs_open is set to 5000
no errors show up in _log
_replicate does not accept POST requests anymore
new replications from _replicator docs do not get picked up
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Jason Smith <jh...@iriscouch.com>.
It's nothing that isn't documented in the Wiki. The hard part is
resisting the temptation to just throw every idea at the problem to
minimize downtime. Even if a service is down (in this case, not every
doc in _replicator was triggering a replication), it usually pays to
deliberately identify the real problem, so you can be sure it's fixed.
Couch is somewhat opaque when you need to troubleshoot (especially ASF
CouchDB which doesn't easily let you get an Erlang shell). But it was
just a matter of noticing tracebacks in the logs.
Limits are there for a reason. Hitting them is not necessarily bad.
Some limits are okay to hard-code to something higher; but I usually
like to make limits (files, ports, quota) a function of something,
such as how many databases you have, or what kind of account you have,
or whatever. Also, when you compute that value, you can trigger an
alert if its value is way too high. (You effectively fail-early,
alerting the sysadmin before there is a crisis.) You might have
guessed that this particular issue did not unfold quite so nicely.
On Thu, Jan 5, 2012 at 10:58 PM, Robert Newson <rn...@apache.org> wrote:
> Hopefully Jason can complete this thread with details of his fix (once
> the sun rolls back around to where he is).
>
> B.
>
> On 5 January 2012 15:37, till <ti...@php.net> wrote:
>> On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gr...@martynus.net>wrote:
>>
>>> I experience massive reliability issues with continuous replications in my
>>> app. Each user has an own profile db with a continuous replication to a
>>> central database. Numbers: ~1800 dbs & ~2600 continuous replications (using
>>> _replicator db), ~200 did not get started.
>>>
>>> My hope is that others faced these issues before and found ways to fix or
>>> work around them. So if you are a developer or know developers that have a
>>> similar couchDB setup, would you mind to get me in touch? Thanks for your
>>> help!
>>>
>>> Tech info:
>>> couchDB 1.1.1 (iriscouch)
>>> max_dbs_open is set to 5000
>>> no errors show up in _log
>>> _replicate does not accept POST requests anymore
>>> new replications from _replicator docs do not get picked up
>>>
>>
>> Is your loglevel at debug? It should def. report something – maybe do a
>> simple cURL request and paste that along with a log entry somewhere (
>> friendpaste.com).
>>
>> You might also want to try stock Apache CouchDB – not sure how IrishCouch's
>> version differs.
>>
>> Till
--
Iris Couch
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Robert Newson <rn...@apache.org>.
Hopefully Jason can complete this thread with details of his fix (once
the sun rolls back around to where he is).
B.
On 5 January 2012 15:37, till <ti...@php.net> wrote:
> On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gr...@martynus.net>wrote:
>
>> I experience massive reliability issues with continuous replications in my
>> app. Each user has an own profile db with a continuous replication to a
>> central database. Numbers: ~1800 dbs & ~2600 continuous replications (using
>> _replicator db), ~200 did not get started.
>>
>> My hope is that others faced these issues before and found ways to fix or
>> work around them. So if you are a developer or know developers that have a
>> similar couchDB setup, would you mind to get me in touch? Thanks for your
>> help!
>>
>> Tech info:
>> couchDB 1.1.1 (iriscouch)
>> max_dbs_open is set to 5000
>> no errors show up in _log
>> _replicate does not accept POST requests anymore
>> new replications from _replicator docs do not get picked up
>>
>
> Is your loglevel at debug? It should def. report something – maybe do a
> simple cURL request and paste that along with a log entry somewhere (
> friendpaste.com).
>
> You might also want to try stock Apache CouchDB – not sure how IrishCouch's
> version differs.
>
> Till
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Gregor Martynus <gr...@martynus.net>.
Till, I'm not sure about the log level, but I'm quite sure that IrisCouch
is using vanilla CouchDB.
I thing the reason, that nothing appeared in the logs was that the requests
didn't even get picked up. They just timed out. But maybe Jason can provide
more information soon.
On Thu, Jan 5, 2012 at 4:37 PM, till <ti...@php.net> wrote:
> On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gregor@martynus.net
> >wrote:
>
> > I experience massive reliability issues with continuous replications in
> my
> > app. Each user has an own profile db with a continuous replication to a
> > central database. Numbers: ~1800 dbs & ~2600 continuous replications
> (using
> > _replicator db), ~200 did not get started.
> >
> > My hope is that others faced these issues before and found ways to fix or
> > work around them. So if you are a developer or know developers that have
> a
> > similar couchDB setup, would you mind to get me in touch? Thanks for your
> > help!
> >
> > Tech info:
> > couchDB 1.1.1 (iriscouch)
> > max_dbs_open is set to 5000
> > no errors show up in _log
> > _replicate does not accept POST requests anymore
> > new replications from _replicator docs do not get picked up
> >
>
> Is your loglevel at debug? It should def. report something – maybe do a
> simple cURL request and paste that along with a log entry somewhere (
> friendpaste.com).
>
> You might also want to try stock Apache CouchDB – not sure how IrishCouch's
> version differs.
>
> Till
>
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by till <ti...@php.net>.
On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gr...@martynus.net>wrote:
> I experience massive reliability issues with continuous replications in my
> app. Each user has an own profile db with a continuous replication to a
> central database. Numbers: ~1800 dbs & ~2600 continuous replications (using
> _replicator db), ~200 did not get started.
>
> My hope is that others faced these issues before and found ways to fix or
> work around them. So if you are a developer or know developers that have a
> similar couchDB setup, would you mind to get me in touch? Thanks for your
> help!
>
> Tech info:
> couchDB 1.1.1 (iriscouch)
> max_dbs_open is set to 5000
> no errors show up in _log
> _replicate does not accept POST requests anymore
> new replications from _replicator docs do not get picked up
>
Is your loglevel at debug? It should def. report something – maybe do a
simple cURL request and paste that along with a log entry somewhere (
friendpaste.com).
You might also want to try stock Apache CouchDB – not sure how IrishCouch's
version differs.
Till
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Gregor Martynus <gr...@martynus.net>.
this is good news Martin, thanks!
Would you mind to provide the numbers of databases and replications you
have at the moment? I really cannot find such numbers anywhere.
On Thu, Jan 5, 2012 at 4:31 PM, Martin Higham <ma...@ocasta.co.uk> wrote:
> Yes, I am working on a similar setup. I ran into replication problems as
> the number of databases increased. I've switched to the 1.2 trunk build as
> replication has been radically modified and seems more reliable.
>
>
> On 5 January 2012 12:53, Gregor Martynus <gr...@martynus.net> wrote:
>
> > Thanks all for your responses! The app is hosted at iris couch, I'm sure
> > that Jason will respond to the technical suggestions, thanks so much!
> >
> > I've a monitoring in place that checks the _log once per minute and
> > notifies me if there is any error. I also checked it manually. There is
> no
> > error. I'm not sure if there are other system logs we could log into, but
> > the logs that are exposed via '/_log' did not show any errors.
> >
> > I'm still wondering if there is anybody else with a similar way to use
> > couchDB as I am, with many databases (e.g. per user) and tons of
> (internal)
> > continuous replications. Is this not a setup that couchDB supports /
> > encourages?
> >
> > On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gregor@martynus.net
> > >wrote:
> >
> > > I experience massive reliability issues with continuous replications in
> > my
> > > app. Each user has an own profile db with a continuous replication to a
> > > central database. Numbers: ~1800 dbs & ~2600 continuous replications
> > > (using _replicator db), ~200 did not get started.
> > >
> > > My hope is that others faced these issues before and found ways to fix
> or
> > > work around them. So if you are a developer or know developers that
> have
> > a
> > > similar couchDB setup, would you mind to get me in touch? Thanks for
> your
> > > help!
> > >
> > > Tech info:
> > > couchDB 1.1.1 (iriscouch)
> > > max_dbs_open is set to 5000
> > > no errors show up in _log
> > > _replicate does not accept POST requests anymore
> > > new replications from _replicator docs do not get picked up
> > >
> >
>
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Martin Higham <ma...@ocasta.co.uk>.
Yes, I am working on a similar setup. I ran into replication problems as
the number of databases increased. I've switched to the 1.2 trunk build as
replication has been radically modified and seems more reliable.
On 5 January 2012 12:53, Gregor Martynus <gr...@martynus.net> wrote:
> Thanks all for your responses! The app is hosted at iris couch, I'm sure
> that Jason will respond to the technical suggestions, thanks so much!
>
> I've a monitoring in place that checks the _log once per minute and
> notifies me if there is any error. I also checked it manually. There is no
> error. I'm not sure if there are other system logs we could log into, but
> the logs that are exposed via '/_log' did not show any errors.
>
> I'm still wondering if there is anybody else with a similar way to use
> couchDB as I am, with many databases (e.g. per user) and tons of (internal)
> continuous replications. Is this not a setup that couchDB supports /
> encourages?
>
> On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gregor@martynus.net
> >wrote:
>
> > I experience massive reliability issues with continuous replications in
> my
> > app. Each user has an own profile db with a continuous replication to a
> > central database. Numbers: ~1800 dbs & ~2600 continuous replications
> > (using _replicator db), ~200 did not get started.
> >
> > My hope is that others faced these issues before and found ways to fix or
> > work around them. So if you are a developer or know developers that have
> a
> > similar couchDB setup, would you mind to get me in touch? Thanks for your
> > help!
> >
> > Tech info:
> > couchDB 1.1.1 (iriscouch)
> > max_dbs_open is set to 5000
> > no errors show up in _log
> > _replicate does not accept POST requests anymore
> > new replications from _replicator docs do not get picked up
> >
>
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Gregor Martynus <gr...@martynus.net>.
Thanks all for your responses! The app is hosted at iris couch, I'm sure
that Jason will respond to the technical suggestions, thanks so much!
I've a monitoring in place that checks the _log once per minute and
notifies me if there is any error. I also checked it manually. There is no
error. I'm not sure if there are other system logs we could log into, but
the logs that are exposed via '/_log' did not show any errors.
I'm still wondering if there is anybody else with a similar way to use
couchDB as I am, with many databases (e.g. per user) and tons of (internal)
continuous replications. Is this not a setup that couchDB supports /
encourages?
On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gr...@martynus.net>wrote:
> I experience massive reliability issues with continuous replications in my
> app. Each user has an own profile db with a continuous replication to a
> central database. Numbers: ~1800 dbs & ~2600 continuous replications
> (using _replicator db), ~200 did not get started.
>
> My hope is that others faced these issues before and found ways to fix or
> work around them. So if you are a developer or know developers that have a
> similar couchDB setup, would you mind to get me in touch? Thanks for your
> help!
>
> Tech info:
> couchDB 1.1.1 (iriscouch)
> max_dbs_open is set to 5000
> no errors show up in _log
> _replicate does not accept POST requests anymore
> new replications from _replicator docs do not get picked up
>
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by till <ti...@php.net>.
On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gr...@martynus.net>wrote:
> I experience massive reliability issues with continuous replications in my
> app. Each user has an own profile db with a continuous replication to a
> central database. Numbers: ~1800 dbs & ~2600 continuous replications (using
> _replicator db), ~200 did not get started.
>
> My hope is that others faced these issues before and found ways to fix or
> work around them. So if you are a developer or know developers that have a
> similar couchDB setup, would you mind to get me in touch? Thanks for your
> help!
>
> Tech info:
> couchDB 1.1.1 (iriscouch)
> max_dbs_open is set to 5000
> no errors show up in _log
> _replicate does not accept POST requests anymore
> new replications from _replicator docs do not get picked up
>
Is your loglevel at debug? It should def. report something – maybe do a
simple cURL request and paste that along with a log entry somewhere (
friendpaste.com).
You might also want to try stock Apache CouchDB – not sure how IrishCouch's
version differs.
Till
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by CGS <cg...@gmail.com>.
That can be also because the database is busy and CouchDB failed to
start new connections due to unresponsive database. Therefore, I think,
it would be quite useful to have a look at the log. If in the log you
get the system limit error, then Robert is right.
CGS
PS: I am not a developer, so, if you aren't interested in my opinion,
please, ignore my post.
On 01/05/2012 10:33 AM, Robert Newson wrote:
> My first thought is to ask if your file descriptor limit is high enough?
>
> On 5 January 2012 09:18, Gregor Martynus<gr...@martynus.net> wrote:
>> I experience massive reliability issues with continuous replications in my
>> app. Each user has an own profile db with a continuous replication to a
>> central database. Numbers: ~1800 dbs& ~2600 continuous replications (using
>> _replicator db), ~200 did not get started.
>>
>> My hope is that others faced these issues before and found ways to fix or
>> work around them. So if you are a developer or know developers that have a
>> similar couchDB setup, would you mind to get me in touch? Thanks for your
>> help!
>>
>> Tech info:
>> couchDB 1.1.1 (iriscouch)
>> max_dbs_open is set to 5000
>> no errors show up in _log
>> _replicate does not accept POST requests anymore
>> new replications from _replicator docs do not get picked up
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Filipe David Manana <fd...@apache.org>.
Check the logs.
Besides the likely issue of not enough available file descriptors for
the connections + database files, it's likely that the Erlang VM
reached the maximum limit of allowed ETS tables (each replication in
1.1.1 and below creates and uses 1 ETS table).
That limit is by default about 1400. To increase it set the
environment variable ERL_MAX_ETS_TABLES to something higher.
On Thu, Jan 5, 2012 at 9:33 AM, Robert Newson <rn...@apache.org> wrote:
> My first thought is to ask if your file descriptor limit is high enough?
>
> On 5 January 2012 09:18, Gregor Martynus <gr...@martynus.net> wrote:
>> I experience massive reliability issues with continuous replications in my
>> app. Each user has an own profile db with a continuous replication to a
>> central database. Numbers: ~1800 dbs & ~2600 continuous replications (using
>> _replicator db), ~200 did not get started.
>>
>> My hope is that others faced these issues before and found ways to fix or
>> work around them. So if you are a developer or know developers that have a
>> similar couchDB setup, would you mind to get me in touch? Thanks for your
>> help!
>>
>> Tech info:
>> couchDB 1.1.1 (iriscouch)
>> max_dbs_open is set to 5000
>> no errors show up in _log
>> _replicate does not accept POST requests anymore
>> new replications from _replicator docs do not get picked up
--
Filipe David Manana,
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Robert Newson <rn...@apache.org>.
My first thought is to ask if your file descriptor limit is high enough?
On 5 January 2012 09:18, Gregor Martynus <gr...@martynus.net> wrote:
> I experience massive reliability issues with continuous replications in my
> app. Each user has an own profile db with a continuous replication to a
> central database. Numbers: ~1800 dbs & ~2600 continuous replications (using
> _replicator db), ~200 did not get started.
>
> My hope is that others faced these issues before and found ways to fix or
> work around them. So if you are a developer or know developers that have a
> similar couchDB setup, would you mind to get me in touch? Thanks for your
> help!
>
> Tech info:
> couchDB 1.1.1 (iriscouch)
> max_dbs_open is set to 5000
> no errors show up in _log
> _replicate does not accept POST requests anymore
> new replications from _replicator docs do not get picked up
Re: couchDB with databases per user: continuous replications stop
working, without errors
Posted by Gregor Martynus <gr...@martynus.net>.
Jason from Iriscouch was able to fix the issue. I just wanted to share his
findings:
As you might have suspected, it was another limit. This will be the
> fourth limit your application hit: first open files in the OS, then
> open database, then ETS tables. If I am correct then we are hitting
> the internal Erlang process limit.
Thanks for all your responses!
On Thu, Jan 5, 2012 at 10:18 AM, Gregor Martynus <gr...@martynus.net>wrote:
> I experience massive reliability issues with continuous replications in my
> app. Each user has an own profile db with a continuous replication to a
> central database. Numbers: ~1800 dbs & ~2600 continuous replications
> (using _replicator db), ~200 did not get started.
>
> My hope is that others faced these issues before and found ways to fix or
> work around them. So if you are a developer or know developers that have a
> similar couchDB setup, would you mind to get me in touch? Thanks for your
> help!
>
> Tech info:
> couchDB 1.1.1 (iriscouch)
> max_dbs_open is set to 5000
> no errors show up in _log
> _replicate does not accept POST requests anymore
> new replications from _replicator docs do not get picked up
>