You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Germain Maurice <ge...@linkfluence.net> on 2010/03/23 10:14:37 UTC
Issues with replication
Hi everybody,
We have a database with more than 8 millions of documents and the size
is more than 450 GB. I wonder if it's a good choice to have all of our
docs in only one file of this size.
We have issues with replication through every time. We tried both one
shot and continuous replication and we got req_timedout followed by a
couchdb crash. We opened reports bugs you can see here :
https://issues.apache.org/jira/browse/COUCHDB-690 and
https://issues.apache.org/jira/browse/COUCHDB-701
Another issue is compaction which doesn't work fine on one of my host.
The compaction is launched on a database which has more than 100 000
documents (10GB) The compaction starts up, works, and after a while,
stops without any warning or alerts. On a other host the same database
was well compacted.
We really wonder how couchdb is used in a production environment. How
many documents do you store ? Do you use big databases or small
databases (file size i mean) ? Embedded replication or not ? Compaction
is efficient or not ?
We are trying to make some developpement in order to avoid the issues we
encounted, we would like some feedback about the best practices in
environment production.
Best regards,
Germain
Re: Issues with replication
Posted by Germain Maurice <ge...@linkfluence.net>.
Hi all,
I still have this kind of compilation problem with apache-couchdb-0.11.0
on Ubuntu 9.10, but it's a bit different from my previous problem :
- With "aptitude install libcurl4-gnutls-dev" :
/bin/bash ../../../libtool --tag=CC --mode=link gcc -g -O2 -lcurl -L/usr/local/lib -L/opt/local/lib -I/usr/local/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -I/usr/include -I/usr/include/js -I/usr/include/mozjs -I/usr/local/include -I/opt/local/include -I/usr/local/include/js -I/opt/local/include/js -DXP_UNIX -lm -o couchjs couchjs-http.o couchjs-main.o couchjs-utf8.o -lcurl -lmozjs -L/usr/local/lib -L/opt/local/lib -lpthread -lcrypt
libtool: link: gcc -g -O2 -I/usr/local/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -I/usr/include -I/usr/include/js -I/usr/include/mozjs -I/usr/local/include -I/opt/local/include -I/usr/local/include/js -I/opt/local/include/js -DXP_UNIX -o couchjs couchjs-http.o couchjs-main.o couchjs-utf8.o -L/usr/local/lib -L/opt/local/lib -lm /usr/lib/libcurl-gnutls.so -lmozjs -lpthread -lcrypt
/usr/lib/libcurl-gnutls.so: undefined reference to `ber_free@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_simple_bind_s@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_next_entry@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_first_attribute@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_memfree@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_get_dn@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_unbind_s@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_msgfree@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_get_values_len@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_set_option@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_free_urldesc@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_url_parse@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_search_s@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_init@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_err2string@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_first_entry@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_next_attribute@OPENLDAP_2.4_2'
/usr/lib/libcurl-gnutls.so: undefined reference to `ldap_value_free_len@OPENLDAP_2.4_2'
My previous compilation problem was :
- With "aptitude install libcurl4-openssl-dev" :
/bin/bash ../../../libtool --tag=CC --mode=link gcc -g -O2 -lcurl -L/usr/local/lib -L/opt/local/lib -I/usr/local/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -I/usr/include -I/usr/include/js -I/usr/include/mozjs -I/usr/local/include -I/opt/local/include -I/usr/local/include/js -I/opt/local/include/js -DXP_UNIX -lm -o couchjs couchjs-http.o couchjs-main.o couchjs-utf8.o -lcurl -lmozjs -L/usr/local/lib -L/opt/local/lib -lpthread -lcrypt
libtool: link: gcc -g -O2 -I/usr/local/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -I/usr/include -I/usr/include/js -I/usr/include/mozjs -I/usr/local/include -I/opt/local/include -I/usr/local/include/js -I/opt/local/include/js -DXP_UNIX -o couchjs couchjs-http.o couchjs-main.o couchjs-utf8.o -L/usr/local/lib -L/opt/local/lib -lm /usr/lib/libcurl.so -lmozjs -lpthread -lcrypt
/usr/lib/libcurl.so: undefined reference to `ldap_set_option@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_get_dn@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_msgfree@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_simple_bind_s@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_next_entry@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_search_s@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_memfree@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ber_free@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_value_free_len@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_next_attribute@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_first_entry@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_err2string@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_free_urldesc@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_url_parse@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_unbind_s@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_first_attribute@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_get_values_len@OPENLDAP_2.4_2'
/usr/lib/libcurl.so: undefined reference to `ldap_init@OPENLDAP_2.4_2'
Le 24/03/10 10:32, Germain Maurice a écrit :
> Hi Randall,
>
> Really thank you for your answer.
>
> I tried to install the release
> http://people.apache.org/~nslater/dist/0.11.0/apache-couchdb-0.11.0.tar.gz
> which is submitted to vote, but the compilation failed on something
> related to ldap. See the attached file.
>
>
> Randall Leeds a écrit :
>> I should add that if you cannot afford to upgrade to 0.11 for whatever
>> reason I have a backport I can provide for 0.10.1.
>>
>> On Tue, Mar 23, 2010 at 13:58, Randall Leeds
>> <ra...@gmail.com> wrote:
>>> Hi Germain,
>>>
>>> Your use case sounds very reasonable and there are similar production
>>> setups in use today. I have seen similar problems in production here
>>> and authored a patch that has made it into 0.11 that should help.
>>> http://issues.apache.org/jira/browse/COUCHDB-597
>>>
>>> However, at least one bug report remains claiming that this does not
>>> fix everything:
>>> https://issues.apache.org/jira/browse/COUCHDB-691
>>>
>>> [...]
>
>
--
Germain Maurice
Administrateur Système/Réseau
Tel : +33.(0)1.42.43.54.33
http://www.linkfluence.net
Re: Issues with replication
Posted by Germain Maurice <ge...@linkfluence.net>.
Hi Randall,
Really thank you for your answer.
I tried to install the release
http://people.apache.org/~nslater/dist/0.11.0/apache-couchdb-0.11.0.tar.gz
which is submitted to vote, but the compilation failed on something
related to ldap. See the attached file.
Randall Leeds a écrit :
> I should add that if you cannot afford to upgrade to 0.11 for whatever
> reason I have a backport I can provide for 0.10.1.
>
> On Tue, Mar 23, 2010 at 13:58, Randall Leeds <ra...@gmail.com> wrote:
>
>> Hi Germain,
>>
>> Your use case sounds very reasonable and there are similar production
>> setups in use today. I have seen similar problems in production here
>> and authored a patch that has made it into 0.11 that should help.
>> http://issues.apache.org/jira/browse/COUCHDB-597
>>
>> However, at least one bug report remains claiming that this does not
>> fix everything:
>> https://issues.apache.org/jira/browse/COUCHDB-691
>>
>> [...]
--
Germain Maurice
Administrateur Système/Réseau
Tel : +33.(0)1.42.43.54.33
http://www.linkfluence.net
**linkfluence news & events**
2009 excellence award nominee from ESOMAR
2009 marketing research silver award from semo & marketing magazine (France)
2009 european excellence award recipient (PR evaluation, wahlradar.de, joint project with Publicis Consultants)
Re: Issues with replication
Posted by Randall Leeds <ra...@gmail.com>.
I should add that if you cannot afford to upgrade to 0.11 for whatever
reason I have a backport I can provide for 0.10.1.
On Tue, Mar 23, 2010 at 13:58, Randall Leeds <ra...@gmail.com> wrote:
> Hi Germain,
>
> Your use case sounds very reasonable and there are similar production
> setups in use today. I have seen similar problems in production here
> and authored a patch that has made it into 0.11 that should help.
> http://issues.apache.org/jira/browse/COUCHDB-597
>
> However, at least one bug report remains claiming that this does not
> fix everything:
> https://issues.apache.org/jira/browse/COUCHDB-691
>
> I'll try to look into it further after this week if no one else gets
> there first. But if you can afford to, please test the 0.11 release
> artifacts (should be stable, currently under vote for release on the
> development list) and re-open COUCHDB-597 or just report your findings
> here.
>
> Find the 0.11 release artifacts here:
> http://mail-archives.apache.org/mod_mbox/couchdb-dev/201003.mbox/%3CDD14BA78-0B61-47CF-A4BE-DFA2D9A47379@me.com%3E
>
> Alternatively, wait a day or two and the vote should (hopefully) pass,
> making it an official release.
>
> -Randall
>
> On Tue, Mar 23, 2010 at 09:41, Germain Maurice
> <ge...@linkfluence.net> wrote:
>> Hi Jan,
>>
>> Yes i forgot to say we are using CouchDB 0.10.1 on Debian 5.0 lenny.
>>
>>
>>
>> Jan Lehnardt a écrit :
>>>
>>> HI Germain,
>>>
>>> which CouchDB version are you running?
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>> On 23 Mar 2010, at 02:14, Germain Maurice wrote:
>>>
>>>
>>>>
>>>> Hi everybody,
>>>>
>>>> We have a database with more than 8 millions of documents and the size is
>>>> more than 450 GB. I wonder if it's a good choice to have all of our docs in
>>>> only one file of this size.
>>>>
>>>> We have issues with replication through every time. We tried both one
>>>> shot and continuous replication and we got req_timedout followed by a
>>>> couchdb crash. We opened reports bugs you can see here :
>>>> https://issues.apache.org/jira/browse/COUCHDB-690 and
>>>> https://issues.apache.org/jira/browse/COUCHDB-701
>>>>
>>>> Another issue is compaction which doesn't work fine on one of my host.
>>>> The compaction is launched on a database which has more than 100 000
>>>> documents (10GB) The compaction starts up, works, and after a while,
>>>> stops without any warning or alerts. On a other host the same database
>>>> was well compacted.
>>>>
>>>> We really wonder how couchdb is used in a production environment. How
>>>> many documents do you store ? Do you use big databases or small
>>>> databases (file size i mean) ? Embedded replication or not ? Compaction
>>>> is efficient or not ?
>>>>
>>>> We are trying to make some developpement in order to avoid the issues we
>>>> encounted, we would like some feedback about the best practices in
>>>> environment production.
>>>>
>>>> Best regards,
>>>> Germain
>>>>
>>>
>>>
>>
>>
>> --
>> Germain Maurice
>> Administrateur Système/Réseau
>> Tel : +33.(0)1.42.43.54.33
>>
>> http://www.linkfluence.net
>> **linkfluence news & events**
>> 2009 excellence award nominee from ESOMAR
>> 2009 marketing research silver award from semo & marketing magazine (France)
>> 2009 european excellence award recipient (PR evaluation, wahlradar.de, joint
>> project with Publicis Consultants)
>>
>>
>
Re: Issues with replication
Posted by Randall Leeds <ra...@gmail.com>.
Hi Germain,
Your use case sounds very reasonable and there are similar production
setups in use today. I have seen similar problems in production here
and authored a patch that has made it into 0.11 that should help.
http://issues.apache.org/jira/browse/COUCHDB-597
However, at least one bug report remains claiming that this does not
fix everything:
https://issues.apache.org/jira/browse/COUCHDB-691
I'll try to look into it further after this week if no one else gets
there first. But if you can afford to, please test the 0.11 release
artifacts (should be stable, currently under vote for release on the
development list) and re-open COUCHDB-597 or just report your findings
here.
Find the 0.11 release artifacts here:
http://mail-archives.apache.org/mod_mbox/couchdb-dev/201003.mbox/%3CDD14BA78-0B61-47CF-A4BE-DFA2D9A47379@me.com%3E
Alternatively, wait a day or two and the vote should (hopefully) pass,
making it an official release.
-Randall
On Tue, Mar 23, 2010 at 09:41, Germain Maurice
<ge...@linkfluence.net> wrote:
> Hi Jan,
>
> Yes i forgot to say we are using CouchDB 0.10.1 on Debian 5.0 lenny.
>
>
>
> Jan Lehnardt a écrit :
>>
>> HI Germain,
>>
>> which CouchDB version are you running?
>>
>> Cheers
>> Jan
>> --
>>
>> On 23 Mar 2010, at 02:14, Germain Maurice wrote:
>>
>>
>>>
>>> Hi everybody,
>>>
>>> We have a database with more than 8 millions of documents and the size is
>>> more than 450 GB. I wonder if it's a good choice to have all of our docs in
>>> only one file of this size.
>>>
>>> We have issues with replication through every time. We tried both one
>>> shot and continuous replication and we got req_timedout followed by a
>>> couchdb crash. We opened reports bugs you can see here :
>>> https://issues.apache.org/jira/browse/COUCHDB-690 and
>>> https://issues.apache.org/jira/browse/COUCHDB-701
>>>
>>> Another issue is compaction which doesn't work fine on one of my host.
>>> The compaction is launched on a database which has more than 100 000
>>> documents (10GB) The compaction starts up, works, and after a while,
>>> stops without any warning or alerts. On a other host the same database
>>> was well compacted.
>>>
>>> We really wonder how couchdb is used in a production environment. How
>>> many documents do you store ? Do you use big databases or small
>>> databases (file size i mean) ? Embedded replication or not ? Compaction
>>> is efficient or not ?
>>>
>>> We are trying to make some developpement in order to avoid the issues we
>>> encounted, we would like some feedback about the best practices in
>>> environment production.
>>>
>>> Best regards,
>>> Germain
>>>
>>
>>
>
>
> --
> Germain Maurice
> Administrateur Système/Réseau
> Tel : +33.(0)1.42.43.54.33
>
> http://www.linkfluence.net
> **linkfluence news & events**
> 2009 excellence award nominee from ESOMAR
> 2009 marketing research silver award from semo & marketing magazine (France)
> 2009 european excellence award recipient (PR evaluation, wahlradar.de, joint
> project with Publicis Consultants)
>
>
Re: Issues with replication
Posted by Germain Maurice <ge...@linkfluence.net>.
Hi Jan,
Yes i forgot to say we are using CouchDB 0.10.1 on Debian 5.0 lenny.
Jan Lehnardt a écrit :
> HI Germain,
>
> which CouchDB version are you running?
>
> Cheers
> Jan
> --
>
> On 23 Mar 2010, at 02:14, Germain Maurice wrote:
>
>
>> Hi everybody,
>>
>> We have a database with more than 8 millions of documents and the size is more than 450 GB. I wonder if it's a good choice to have all of our docs in only one file of this size.
>>
>> We have issues with replication through every time. We tried both one shot and continuous replication and we got req_timedout followed by a couchdb crash. We opened reports bugs you can see here : https://issues.apache.org/jira/browse/COUCHDB-690 and
>> https://issues.apache.org/jira/browse/COUCHDB-701
>>
>> Another issue is compaction which doesn't work fine on one of my host.
>> The compaction is launched on a database which has more than 100 000
>> documents (10GB) The compaction starts up, works, and after a while,
>> stops without any warning or alerts. On a other host the same database
>> was well compacted.
>>
>> We really wonder how couchdb is used in a production environment. How
>> many documents do you store ? Do you use big databases or small
>> databases (file size i mean) ? Embedded replication or not ? Compaction
>> is efficient or not ?
>>
>> We are trying to make some developpement in order to avoid the issues we
>> encounted, we would like some feedback about the best practices in environment production.
>>
>> Best regards,
>> Germain
>>
>
>
--
Germain Maurice
Administrateur Système/Réseau
Tel : +33.(0)1.42.43.54.33
http://www.linkfluence.net
**linkfluence news & events**
2009 excellence award nominee from ESOMAR
2009 marketing research silver award from semo & marketing magazine (France)
2009 european excellence award recipient (PR evaluation, wahlradar.de, joint project with Publicis Consultants)
Re: Issues with replication
Posted by Jan Lehnardt <ja...@apache.org>.
HI Germain,
which CouchDB version are you running?
Cheers
Jan
--
On 23 Mar 2010, at 02:14, Germain Maurice wrote:
> Hi everybody,
>
> We have a database with more than 8 millions of documents and the size is more than 450 GB. I wonder if it's a good choice to have all of our docs in only one file of this size.
>
> We have issues with replication through every time. We tried both one shot and continuous replication and we got req_timedout followed by a couchdb crash. We opened reports bugs you can see here : https://issues.apache.org/jira/browse/COUCHDB-690 and
> https://issues.apache.org/jira/browse/COUCHDB-701
>
> Another issue is compaction which doesn't work fine on one of my host.
> The compaction is launched on a database which has more than 100 000
> documents (10GB) The compaction starts up, works, and after a while,
> stops without any warning or alerts. On a other host the same database
> was well compacted.
>
> We really wonder how couchdb is used in a production environment. How
> many documents do you store ? Do you use big databases or small
> databases (file size i mean) ? Embedded replication or not ? Compaction
> is efficient or not ?
>
> We are trying to make some developpement in order to avoid the issues we
> encounted, we would like some feedback about the best practices in environment production.
>
> Best regards,
> Germain