Optimizing view building

[Nadav Samet <th...@gmail.com>]

Hi,

I have a 600k documents in one database and building a view (with a trivial
map function) takes about 40 minutes on a single core server. I was
wondering if there's a way to tell how the time is divided between running
the javascript map function and actually building the view.


-- 
Sincerely yours,
Nadav

Re: Optimizing view building

[Adam Kocoloski <ko...@apache.org>]

On Jun 9, 2009, at 5:24 AM, Brian Candler wrote:

> But I'm hoping that some sort of batching of view processing would  
> make as
> big a difference to view performance as it does when bulk loading.  
> (Perhaps
> send 1MB of documents at a time to the view server?)

We already do this, with a 10MB process memory threshold.  Best,

Adam

Re: Optimizing view building

[Brian Candler <B....@pobox.com>]

On Sat, Jun 06, 2009 at 12:18:34AM +0200, Jan Lehnardt wrote:
> beam & couchjs wait for each other while the respective partner process
> is working. There is room for optimization, we haven't gotten around to
> that yet :)
>
> See this and the dev@ mailing list archives for detailed discussions.

I would very much like to see this. I have a small test database with only
~1,400 documents, expanding to maybe ~12,000 key/value pairs. On my 1.2GHz
laptop it takes only about 2 seconds to bulk-load these documents, but over
90 seconds to index them.

My first job will be to turn this into a proper reproducible benchmark, and
then to see how much erlview improves it.

But I'm hoping that some sort of batching of view processing would make as
big a difference to view performance as it does when bulk loading. (Perhaps
send 1MB of documents at a time to the view server?)

Regards,

Brian.

Re: Optimizing view building

[Jan Lehnardt <ja...@apache.org>]

On 6 Jun 2009, at 00:05, Randall Leeds wrote:

> On Fri, Jun 5, 2009 at 16:23, Nadav Samet <th...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a 600k documents in one database and building a view (with a  
>> trivial
>> map function) takes about 40 minutes on a single core server. I was
>> wondering if there's a way to tell how the time is divided between  
>> running
>> the javascript map function and actually building the view.
>
>
> The javascript runs in a separate process so you could watch the CPU  
> time
> for beam vs the couchjs process.

beam & couchjs wait for each other while the respective partner process
is working. There is room for optimization, we haven't gotten around to
that yet :)

See this and the dev@ mailing list archives for detailed discussions.

Cheers
Jan
--

Re: Optimizing view building

[Randall Leeds <ra...@gmail.com>]

On Fri, Jun 5, 2009 at 16:23, Nadav Samet <th...@gmail.com> wrote:

> Hi,
>
> I have a 600k documents in one database and building a view (with a trivial
> map function) takes about 40 minutes on a single core server. I was
> wondering if there's a way to tell how the time is divided between running
> the javascript map function and actually building the view.


The javascript runs in a separate process so you could watch the CPU time
for beam vs the couchjs process.