You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Huw Selley <hu...@netdev.co.uk> on 2010/12/09 13:32:10 UTC

view building batch sizes

Hi,

I read on http://guide.couchdb.org/draft/performance.html that 

"Views load a batch of updates from disk, pass them through the view engine, and then write the view rows out. Each batch is a few hundred documents, so the writer can take advantage of the bulk efficiencies we see in the next section."

Is there a method to change the batch size? I would like to try measure the impact of using smaller and larger batches.

Thanks in advance
Huw

Re: view building batch sizes

Posted by Adam Kocoloski <ko...@apache.org>.
On Dec 9, 2010, at 10:49 AM, Paul Davis wrote:

> On Thu, Dec 9, 2010 at 10:47 AM, Jan Lehnardt <ja...@apache.org> wrote:
>> 
>> On 9 Dec 2010, at 15:37, Paul Davis wrote:
>> 
>>> On Thu, Dec 9, 2010 at 7:51 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>>> Hi Huw,
>>>> 
>>>> 
>>>> On 9 Dec 2010, at 13:32, Huw Selley wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I read on http://guide.couchdb.org/draft/performance.html that
>>>>> 
>>>>> "Views load a batch of updates from disk, pass them through the view engine, and then write the view rows out. Each batch is a few hundred documents, so the writer can take advantage of the bulk efficiencies we see in the next section."
>>>>> 
>>>>> Is there a method to change the batch size? I would like to try measure the impact of using smaller and larger batches.
>>>> 
>>>> Thanks for helping to profile things. You may want to take this to
>>>> dev@couchdb.apache.org as it is the development-related mailing list.
>>>> 
>>>> For tuning these values, see src/couchdb/couch_view_updater.erl
>>>> 
>>>> The `update()` function has these lines:
>>>> 
>>>>    {ok, MapQueue} = couch_work_queue:new(100000, 500),
>>>>    {ok, WriteQueue} = couch_work_queue:new(100000, 500),
>>>> 
>>>> They set up a queue for mapping and writing each. The parameters are
>>>> 
>>>>    couch_work_queue:new(MaxSize, MaxItems)
>>>> 
>>>> If either maximum is hit, the queue is deemed full.
>>>> 
>>>> Note: This is from about 30 seconds of looking at the source, so I
>>>> might miss a subtlety or three.
>>>> 
>>>> Cheers
>>>> Jan
>>>> --
>>>> 
>>>> 
>>>> 
>>> 
>>> The only real subtlety is that we don't wait for a minimum amount to
>>> be inserted into the queue. Playing with larger or smaller queues on
>>> either side might be an interesting bit. Also, for testing it might
>>> not be a bad idea to add config values for these values.
>> 
>> 
>> Good thinking, I made a patch:
>> 
>>  https://github.com/janl/couchdb/commit/547691a9f4b9895086f2763af84e1cc459e4d72c
>> 
>> Branch:
>> 
>>  https://github.com/janl/couchdb/tree/config-view-batches
>> 
>> "Compiles for me".
>> 
>> To make this proper, we probably want to move the lookups into
>> couch_view_group:init/3 and pass the values down, but it should
>> be ok as is.
>> 
> 
> Probably not worth it. ets looks like that are quick, and as a
> percentage of a view build are going to be fairly inconsequential.
> Though, you could make a case about coupling.

This issue was initially raised in https://issues.apache.org/jira/browse/COUCHDB-700 .  I'm not opposed to making the queue sizes configurable, but I think the more important fix by far is to be able to configure a minimum number of items in the work unit sent to the reducer.  Cheers,

Adam

Re: view building batch sizes

Posted by Paul Davis <pa...@gmail.com>.
On Thu, Dec 9, 2010 at 10:47 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On 9 Dec 2010, at 15:37, Paul Davis wrote:
>
>> On Thu, Dec 9, 2010 at 7:51 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>> Hi Huw,
>>>
>>>
>>> On 9 Dec 2010, at 13:32, Huw Selley wrote:
>>>
>>>> Hi,
>>>>
>>>> I read on http://guide.couchdb.org/draft/performance.html that
>>>>
>>>> "Views load a batch of updates from disk, pass them through the view engine, and then write the view rows out. Each batch is a few hundred documents, so the writer can take advantage of the bulk efficiencies we see in the next section."
>>>>
>>>> Is there a method to change the batch size? I would like to try measure the impact of using smaller and larger batches.
>>>
>>> Thanks for helping to profile things. You may want to take this to
>>> dev@couchdb.apache.org as it is the development-related mailing list.
>>>
>>> For tuning these values, see src/couchdb/couch_view_updater.erl
>>>
>>> The `update()` function has these lines:
>>>
>>>    {ok, MapQueue} = couch_work_queue:new(100000, 500),
>>>    {ok, WriteQueue} = couch_work_queue:new(100000, 500),
>>>
>>> They set up a queue for mapping and writing each. The parameters are
>>>
>>>    couch_work_queue:new(MaxSize, MaxItems)
>>>
>>> If either maximum is hit, the queue is deemed full.
>>>
>>> Note: This is from about 30 seconds of looking at the source, so I
>>> might miss a subtlety or three.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>>
>>>
>>
>> The only real subtlety is that we don't wait for a minimum amount to
>> be inserted into the queue. Playing with larger or smaller queues on
>> either side might be an interesting bit. Also, for testing it might
>> not be a bad idea to add config values for these values.
>
>
> Good thinking, I made a patch:
>
>  https://github.com/janl/couchdb/commit/547691a9f4b9895086f2763af84e1cc459e4d72c
>
> Branch:
>
>  https://github.com/janl/couchdb/tree/config-view-batches
>
> "Compiles for me".
>
> To make this proper, we probably want to move the lookups into
> couch_view_group:init/3 and pass the values down, but it should
> be ok as is.
>

Probably not worth it. ets looks like that are quick, and as a
percentage of a view build are going to be fairly inconsequential.
Though, you could make a case about coupling.

> Cheers
> Jan
> --
>
>

Re: view building batch sizes

Posted by Jan Lehnardt <ja...@apache.org>.
On 9 Dec 2010, at 15:37, Paul Davis wrote:

> On Thu, Dec 9, 2010 at 7:51 AM, Jan Lehnardt <ja...@apache.org> wrote:
>> Hi Huw,
>> 
>> 
>> On 9 Dec 2010, at 13:32, Huw Selley wrote:
>> 
>>> Hi,
>>> 
>>> I read on http://guide.couchdb.org/draft/performance.html that
>>> 
>>> "Views load a batch of updates from disk, pass them through the view engine, and then write the view rows out. Each batch is a few hundred documents, so the writer can take advantage of the bulk efficiencies we see in the next section."
>>> 
>>> Is there a method to change the batch size? I would like to try measure the impact of using smaller and larger batches.
>> 
>> Thanks for helping to profile things. You may want to take this to
>> dev@couchdb.apache.org as it is the development-related mailing list.
>> 
>> For tuning these values, see src/couchdb/couch_view_updater.erl
>> 
>> The `update()` function has these lines:
>> 
>>    {ok, MapQueue} = couch_work_queue:new(100000, 500),
>>    {ok, WriteQueue} = couch_work_queue:new(100000, 500),
>> 
>> They set up a queue for mapping and writing each. The parameters are
>> 
>>    couch_work_queue:new(MaxSize, MaxItems)
>> 
>> If either maximum is hit, the queue is deemed full.
>> 
>> Note: This is from about 30 seconds of looking at the source, so I
>> might miss a subtlety or three.
>> 
>> Cheers
>> Jan
>> --
>> 
>> 
>> 
> 
> The only real subtlety is that we don't wait for a minimum amount to
> be inserted into the queue. Playing with larger or smaller queues on
> either side might be an interesting bit. Also, for testing it might
> not be a bad idea to add config values for these values.


Good thinking, I made a patch:

  https://github.com/janl/couchdb/commit/547691a9f4b9895086f2763af84e1cc459e4d72c

Branch:

 https://github.com/janl/couchdb/tree/config-view-batches

"Compiles for me".

To make this proper, we probably want to move the lookups into 
couch_view_group:init/3 and pass the values down, but it should
be ok as is.

Cheers
Jan
-- 


Re: view building batch sizes

Posted by Paul Davis <pa...@gmail.com>.
On Thu, Dec 9, 2010 at 7:51 AM, Jan Lehnardt <ja...@apache.org> wrote:
> Hi Huw,
>
>
> On 9 Dec 2010, at 13:32, Huw Selley wrote:
>
>> Hi,
>>
>> I read on http://guide.couchdb.org/draft/performance.html that
>>
>> "Views load a batch of updates from disk, pass them through the view engine, and then write the view rows out. Each batch is a few hundred documents, so the writer can take advantage of the bulk efficiencies we see in the next section."
>>
>> Is there a method to change the batch size? I would like to try measure the impact of using smaller and larger batches.
>
> Thanks for helping to profile things. You may want to take this to
> dev@couchdb.apache.org as it is the development-related mailing list.
>
> For tuning these values, see src/couchdb/couch_view_updater.erl
>
> The `update()` function has these lines:
>
>    {ok, MapQueue} = couch_work_queue:new(100000, 500),
>    {ok, WriteQueue} = couch_work_queue:new(100000, 500),
>
> They set up a queue for mapping and writing each. The parameters are
>
>    couch_work_queue:new(MaxSize, MaxItems)
>
> If either maximum is hit, the queue is deemed full.
>
> Note: This is from about 30 seconds of looking at the source, so I
> might miss a subtlety or three.
>
> Cheers
> Jan
> --
>
>
>

The only real subtlety is that we don't wait for a minimum amount to
be inserted into the queue. Playing with larger or smaller queues on
either side might be an interesting bit. Also, for testing it might
not be a bad idea to add config values for these values.

Paul

Re: view building batch sizes

Posted by Jan Lehnardt <ja...@apache.org>.
Hi Huw,


On 9 Dec 2010, at 13:32, Huw Selley wrote:

> Hi,
> 
> I read on http://guide.couchdb.org/draft/performance.html that 
> 
> "Views load a batch of updates from disk, pass them through the view engine, and then write the view rows out. Each batch is a few hundred documents, so the writer can take advantage of the bulk efficiencies we see in the next section."
> 
> Is there a method to change the batch size? I would like to try measure the impact of using smaller and larger batches.

Thanks for helping to profile things. You may want to take this to
dev@couchdb.apache.org as it is the development-related mailing list.

For tuning these values, see src/couchdb/couch_view_updater.erl

The `update()` function has these lines:

    {ok, MapQueue} = couch_work_queue:new(100000, 500),
    {ok, WriteQueue} = couch_work_queue:new(100000, 500),

They set up a queue for mapping and writing each. The parameters are

    couch_work_queue:new(MaxSize, MaxItems)

If either maximum is hit, the queue is deemed full.

Note: This is from about 30 seconds of looking at the source, so I 
might miss a subtlety or three.

Cheers
Jan
--