You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Vladimir Ralev <vl...@gmail.com> on 2014/01/22 19:17:56 UTC
Compaction of views deletes the indices?
Hi guys,
I am experimenting with compaction ops. In a large DB with many big views a
compaction renders the DB unusable for almost 10 hours or so while I force
each view to be loaded by a script. Mass indexing completely overwhelms the
system.
Is there some way to do compaction without losing the view indices? The way
I understand it indexing and compaction should be completely separate.
Re: Compaction of views deletes the indices?
Posted by Robert Samuel Newson <rn...@apache.org>.
Ah, you didn’t say this was not CouchDB. BigCouch does not have that fix.
Verify that you have truly raised file descriptor limit by looking at /proc/`pidof beam.smp`/limits on all nodes.
1,000’s of compactions at once is unlikely to ever work, you’ll need to manage a smaller load than that. BigCouch, like CouchDB, runs compaction at the same priority as other tasks.
B.
On 23 Jan 2014, at 17:21, Vladimir Ralev <vl...@gmail.com> wrote:
> Thanks. I do see emfail errors all over the place indeed. As a workaround
> is it feasible to enable some large system limits. I already have 1M linux
> file handles limit. I am forcing compaction on a large scale - 1000s of
> databases further segmented by bigcouch. I will try to pace my script next
> time i run it hoping it wouldn't open all files at the same time, but still
> not sure if I should touch other settings inside erlang itself.
>
>
> On Thu, Jan 23, 2014 at 10:10 AM, Robert Samuel Newson
> <rn...@apache.org>wrote:
>
>>
>> View indexes are not lost when compacting, but it’s possible you’re a
>> release with bug COUCHDB-1445 in it (
>> https://issues.apache.org/jira/browse/COUCHDB-1445). This was fixed in
>> 1.2.0 and up.
>>
>> B.
>>
>> On 22 Jan 2014, at 18:17, Vladimir Ralev <vl...@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I am experimenting with compaction ops. In a large DB with many big
>> views a
>>> compaction renders the DB unusable for almost 10 hours or so while I
>> force
>>> each view to be loaded by a script. Mass indexing completely overwhelms
>> the
>>> system.
>>>
>>> Is there some way to do compaction without losing the view indices? The
>> way
>>> I understand it indexing and compaction should be completely separate.
>>
>>
Re: Compaction of views deletes the indices?
Posted by Vladimir Ralev <vl...@gmail.com>.
Thanks. I do see emfail errors all over the place indeed. As a workaround
is it feasible to enable some large system limits. I already have 1M linux
file handles limit. I am forcing compaction on a large scale - 1000s of
databases further segmented by bigcouch. I will try to pace my script next
time i run it hoping it wouldn't open all files at the same time, but still
not sure if I should touch other settings inside erlang itself.
On Thu, Jan 23, 2014 at 10:10 AM, Robert Samuel Newson
<rn...@apache.org>wrote:
>
> View indexes are not lost when compacting, but it’s possible you’re a
> release with bug COUCHDB-1445 in it (
> https://issues.apache.org/jira/browse/COUCHDB-1445). This was fixed in
> 1.2.0 and up.
>
> B.
>
> On 22 Jan 2014, at 18:17, Vladimir Ralev <vl...@gmail.com> wrote:
>
> > Hi guys,
> >
> > I am experimenting with compaction ops. In a large DB with many big
> views a
> > compaction renders the DB unusable for almost 10 hours or so while I
> force
> > each view to be loaded by a script. Mass indexing completely overwhelms
> the
> > system.
> >
> > Is there some way to do compaction without losing the view indices? The
> way
> > I understand it indexing and compaction should be completely separate.
>
>
Re: Compaction of views deletes the indices?
Posted by Robert Samuel Newson <rn...@apache.org>.
View indexes are not lost when compacting, but it’s possible you’re a release with bug COUCHDB-1445 in it (https://issues.apache.org/jira/browse/COUCHDB-1445). This was fixed in 1.2.0 and up.
B.
On 22 Jan 2014, at 18:17, Vladimir Ralev <vl...@gmail.com> wrote:
> Hi guys,
>
> I am experimenting with compaction ops. In a large DB with many big views a
> compaction renders the DB unusable for almost 10 hours or so while I force
> each view to be loaded by a script. Mass indexing completely overwhelms the
> system.
>
> Is there some way to do compaction without losing the view indices? The way
> I understand it indexing and compaction should be completely separate.