You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by ev...@bath.ac.uk on 2010/11/16 14:00:51 UTC
Large attachments
Hi everyone,
I'm trying to work with some large attachments (around 1.5 GB). When I
go to download these (as a standalone attachment) the CouchDB process
grows in size by at least the size of the attachment before the
download starts. This implies that the attachment is being loaded into
memory entirely before being sent to the client. Has anyone else seen
this behaviour? Is this a bug, or is there a configuration change I
can make to resolve this?
I've tried disabling compression on attachments in case it's the
compression that's causing the problem.
I'm using 1.0.1.
Thanks, David
Re: Large attachments
Posted by Nils Breunese <N....@vpro.nl>.
Bram Neijt wrote:
> Run
> for i in {0..50};do curl http://localhost:5984/[test
> database]/[doc_id]/test.bin > /dev/null 2>&1 & done
>
> This will create 50 curl processes which download from your couchdb.
51 actually. :o)
Nils.
------------------------------------------------------------------------
VPRO
phone: +31(0)356712911
e-mail: info@vpro.nl
web: www.vpro.nl
------------------------------------------------------------------------
Re: Large attachments
Posted by Robert Newson <ro...@gmail.com>.
David,
I've failed to reproduce this locally by following your instructions.
My memory usage was stable (OS X). Another user has tried the test on
Linux with R13 and reports stable memory usage also.
Can you provide more details of the OS, hardware and the manner in
which you are monitoring the memory usage itself? I'd like to
eliminate as many distracting factors as possible.
Thanks,
B.
On Thu, Nov 25, 2010 at 11:01 AM, <ev...@bath.ac.uk> wrote:
> Quoting Benoit Chesneau <bc...@gmail.com>:
>
>> On Mon, Nov 22, 2010 at 3:51 PM, Bram Neijt <bn...@gmail.com> wrote:
>>>
>>> Bit of a mis-understanding here, it is about downloads, not uploads.
>>>
>>> For example:
>>> dd if=/dev/urandom of=/tmp/test.bin count=50000 bs=10240
>>> Put test.bin as an attachment in a coucdb database
>>> Run
>>> for i in {0..50};do curl http://localhost:5984/[test
>>> database]/[doc_id]/test.bin > /dev/null 2>&1 & done
>>>
>>> This will create 50 curl processes which download from your couchdb.
>>> Looking at the memory consumption of couchdb, it seems like it is
>>> loading large parts of the file into memory.
>>>
>>> Bram
>>>
>> what is the exact memory usage ?
>
> The process appears to grow by approximately the attachment size, per client
> connection implying that the entire attachment is being buffered before
> being sent to the client.
>
> I've raised this as an issue:
> https://issues.apache.org/jira/browse/COUCHDB-964
>
> David
>
>
>
Re: Large attachments
Posted by ev...@bath.ac.uk.
Quoting Benoit Chesneau <bc...@gmail.com>:
> On Mon, Nov 22, 2010 at 3:51 PM, Bram Neijt <bn...@gmail.com> wrote:
>> Bit of a mis-understanding here, it is about downloads, not uploads.
>>
>> For example:
>> dd if=/dev/urandom of=/tmp/test.bin count=50000 bs=10240
>> Put test.bin as an attachment in a coucdb database
>> Run
>> for i in {0..50};do curl http://localhost:5984/[test
>> database]/[doc_id]/test.bin > /dev/null 2>&1 & done
>>
>> This will create 50 curl processes which download from your couchdb.
>> Looking at the memory consumption of couchdb, it seems like it is
>> loading large parts of the file into memory.
>>
>> Bram
>>
> what is the exact memory usage ?
The process appears to grow by approximately the attachment size, per
client connection implying that the entire attachment is being
buffered before being sent to the client.
I've raised this as an issue:
https://issues.apache.org/jira/browse/COUCHDB-964
David
Re: Large attachments
Posted by Benoit Chesneau <bc...@gmail.com>.
On Mon, Nov 22, 2010 at 3:51 PM, Bram Neijt <bn...@gmail.com> wrote:
> Bit of a mis-understanding here, it is about downloads, not uploads.
>
> For example:
> dd if=/dev/urandom of=/tmp/test.bin count=50000 bs=10240
> Put test.bin as an attachment in a coucdb database
> Run
> for i in {0..50};do curl http://localhost:5984/[test
> database]/[doc_id]/test.bin > /dev/null 2>&1 & done
>
> This will create 50 curl processes which download from your couchdb.
> Looking at the memory consumption of couchdb, it seems like it is
> loading large parts of the file into memory.
>
> Bram
>
what is the exact memory usage ?
- benoît
Re: Large attachments
Posted by Jan Lehnardt <ja...@apache.org>.
On 22 Nov 2010, at 15:51, Bram Neijt wrote:
> Bit of a mis-understanding here, it is about downloads, not uploads.
>
> For example:
> dd if=/dev/urandom of=/tmp/test.bin count=50000 bs=10240
> Put test.bin as an attachment in a coucdb database
> Run
> for i in {0..50};do curl http://localhost:5984/[test
> database]/[doc_id]/test.bin > /dev/null 2>&1 & done
>
> This will create 50 curl processes which download from your couchdb.
> Looking at the memory consumption of couchdb, it seems like it is
> loading large parts of the file into memory.
Curious. Can you open a JIRA ticket for this?
https://issues.apache.org/jira/browse/COUCHDB
Cheers
Jan
--
>
> Bram
>
>
> On Mon, Nov 22, 2010 at 3:11 PM, Robert Newson <ro...@gmail.com> wrote:
>> Curl buffers binary uploads, depending on the manner you perform the operation.
>>
>> B.
>>
>> On Mon, Nov 22, 2010 at 2:03 PM, Bram Neijt <bn...@gmail.com> wrote:
>>> I can reproduce this problem: if I upload a 500 MB and start 10
>>> concurrent curl commands, memory usage increase dramatically with the
>>> following environment:
>>> Description: Ubuntu 10.10
>>> Release: 10.10
>>> Codename: maverick
>>> {"couchdb":"Welcome","version":"1.0.1"}
>>>
>>> Bram
>>>
>>> On Tue, Nov 16, 2010 at 5:56 PM, <ev...@bath.ac.uk> wrote:
>>>> Well, I'm just doing a GET directly to the document_id + attachment:
>>>> http://localhost:5984/database/doc_id/attachment
>>>>
>>>> Clicking on the attachment in Futon would have the same effect.
>>>>
>>>> David
>>>>
>>>> Quoting Jan Lehnardt <ja...@apache.org>:
>>>>
>>>>> Hi David,
>>>>>
>>>>> On 16 Nov 2010, at 14:00, evxdo@bath.ac.uk wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I'm trying to work with some large attachments (around 1.5 GB). When I
>>>>>> go to download these (as a standalone attachment) the CouchDB process grows
>>>>>> in size by at least the size of the attachment before the download starts.
>>>>>> This implies that the attachment is being loaded into memory entirely
>>>>>> before being sent to the client. Has anyone else seen this behaviour? Is
>>>>>> this a bug, or is there a configuration change I can make to resolve this?
>>>>>>
>>>>>> I've tried disabling compression on attachments in case it's the
>>>>>> compression that's causing the problem.
>>>>>>
>>>>>> I'm using 1.0.1.
>>>>>
>>>>> What does your request look like?
>>>>>
>>>>> The standalone attachment API does not buffer.
>>>>>
>>>>> Cheers
>>>>> Jan
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
Re: Large attachments
Posted by Bram Neijt <bn...@gmail.com>.
Bit of a mis-understanding here, it is about downloads, not uploads.
For example:
dd if=/dev/urandom of=/tmp/test.bin count=50000 bs=10240
Put test.bin as an attachment in a coucdb database
Run
for i in {0..50};do curl http://localhost:5984/[test
database]/[doc_id]/test.bin > /dev/null 2>&1 & done
This will create 50 curl processes which download from your couchdb.
Looking at the memory consumption of couchdb, it seems like it is
loading large parts of the file into memory.
Bram
On Mon, Nov 22, 2010 at 3:11 PM, Robert Newson <ro...@gmail.com> wrote:
> Curl buffers binary uploads, depending on the manner you perform the operation.
>
> B.
>
> On Mon, Nov 22, 2010 at 2:03 PM, Bram Neijt <bn...@gmail.com> wrote:
>> I can reproduce this problem: if I upload a 500 MB and start 10
>> concurrent curl commands, memory usage increase dramatically with the
>> following environment:
>> Description: Ubuntu 10.10
>> Release: 10.10
>> Codename: maverick
>> {"couchdb":"Welcome","version":"1.0.1"}
>>
>> Bram
>>
>> On Tue, Nov 16, 2010 at 5:56 PM, <ev...@bath.ac.uk> wrote:
>>> Well, I'm just doing a GET directly to the document_id + attachment:
>>> http://localhost:5984/database/doc_id/attachment
>>>
>>> Clicking on the attachment in Futon would have the same effect.
>>>
>>> David
>>>
>>> Quoting Jan Lehnardt <ja...@apache.org>:
>>>
>>>> Hi David,
>>>>
>>>> On 16 Nov 2010, at 14:00, evxdo@bath.ac.uk wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I'm trying to work with some large attachments (around 1.5 GB). When I
>>>>> go to download these (as a standalone attachment) the CouchDB process grows
>>>>> in size by at least the size of the attachment before the download starts.
>>>>> This implies that the attachment is being loaded into memory entirely
>>>>> before being sent to the client. Has anyone else seen this behaviour? Is
>>>>> this a bug, or is there a configuration change I can make to resolve this?
>>>>>
>>>>> I've tried disabling compression on attachments in case it's the
>>>>> compression that's causing the problem.
>>>>>
>>>>> I'm using 1.0.1.
>>>>
>>>> What does your request look like?
>>>>
>>>> The standalone attachment API does not buffer.
>>>>
>>>> Cheers
>>>> Jan
>>>> --
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>
Re: Large attachments
Posted by Robert Newson <ro...@gmail.com>.
Curl buffers binary uploads, depending on the manner you perform the operation.
B.
On Mon, Nov 22, 2010 at 2:03 PM, Bram Neijt <bn...@gmail.com> wrote:
> I can reproduce this problem: if I upload a 500 MB and start 10
> concurrent curl commands, memory usage increase dramatically with the
> following environment:
> Description: Ubuntu 10.10
> Release: 10.10
> Codename: maverick
> {"couchdb":"Welcome","version":"1.0.1"}
>
> Bram
>
> On Tue, Nov 16, 2010 at 5:56 PM, <ev...@bath.ac.uk> wrote:
>> Well, I'm just doing a GET directly to the document_id + attachment:
>> http://localhost:5984/database/doc_id/attachment
>>
>> Clicking on the attachment in Futon would have the same effect.
>>
>> David
>>
>> Quoting Jan Lehnardt <ja...@apache.org>:
>>
>>> Hi David,
>>>
>>> On 16 Nov 2010, at 14:00, evxdo@bath.ac.uk wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I'm trying to work with some large attachments (around 1.5 GB). When I
>>>> go to download these (as a standalone attachment) the CouchDB process grows
>>>> in size by at least the size of the attachment before the download starts.
>>>> This implies that the attachment is being loaded into memory entirely
>>>> before being sent to the client. Has anyone else seen this behaviour? Is
>>>> this a bug, or is there a configuration change I can make to resolve this?
>>>>
>>>> I've tried disabling compression on attachments in case it's the
>>>> compression that's causing the problem.
>>>>
>>>> I'm using 1.0.1.
>>>
>>> What does your request look like?
>>>
>>> The standalone attachment API does not buffer.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>>
>>>
>>
>>
>>
>>
>
Re: Large attachments
Posted by Bram Neijt <bn...@gmail.com>.
I can reproduce this problem: if I upload a 500 MB and start 10
concurrent curl commands, memory usage increase dramatically with the
following environment:
Description: Ubuntu 10.10
Release: 10.10
Codename: maverick
{"couchdb":"Welcome","version":"1.0.1"}
Bram
On Tue, Nov 16, 2010 at 5:56 PM, <ev...@bath.ac.uk> wrote:
> Well, I'm just doing a GET directly to the document_id + attachment:
> http://localhost:5984/database/doc_id/attachment
>
> Clicking on the attachment in Futon would have the same effect.
>
> David
>
> Quoting Jan Lehnardt <ja...@apache.org>:
>
>> Hi David,
>>
>> On 16 Nov 2010, at 14:00, evxdo@bath.ac.uk wrote:
>>
>>> Hi everyone,
>>>
>>> I'm trying to work with some large attachments (around 1.5 GB). When I
>>> go to download these (as a standalone attachment) the CouchDB process grows
>>> in size by at least the size of the attachment before the download starts.
>>> This implies that the attachment is being loaded into memory entirely
>>> before being sent to the client. Has anyone else seen this behaviour? Is
>>> this a bug, or is there a configuration change I can make to resolve this?
>>>
>>> I've tried disabling compression on attachments in case it's the
>>> compression that's causing the problem.
>>>
>>> I'm using 1.0.1.
>>
>> What does your request look like?
>>
>> The standalone attachment API does not buffer.
>>
>> Cheers
>> Jan
>> --
>>
>>
>>
>
>
>
>
Re: Large attachments
Posted by ev...@bath.ac.uk.
Well, I'm just doing a GET directly to the document_id + attachment:
http://localhost:5984/database/doc_id/attachment
Clicking on the attachment in Futon would have the same effect.
David
Quoting Jan Lehnardt <ja...@apache.org>:
> Hi David,
>
> On 16 Nov 2010, at 14:00, evxdo@bath.ac.uk wrote:
>
>> Hi everyone,
>>
>> I'm trying to work with some large attachments (around 1.5 GB).
>> When I go to download these (as a standalone attachment) the
>> CouchDB process grows in size by at least the size of the
>> attachment before the download starts. This implies that the
>> attachment is being loaded into memory entirely before being sent
>> to the client. Has anyone else seen this behaviour? Is this a bug,
>> or is there a configuration change I can make to resolve this?
>>
>> I've tried disabling compression on attachments in case it's the
>> compression that's causing the problem.
>>
>> I'm using 1.0.1.
>
> What does your request look like?
>
> The standalone attachment API does not buffer.
>
> Cheers
> Jan
> --
>
>
>
Re: Large attachments
Posted by Jan Lehnardt <ja...@apache.org>.
Hi David,
On 16 Nov 2010, at 14:00, evxdo@bath.ac.uk wrote:
> Hi everyone,
>
> I'm trying to work with some large attachments (around 1.5 GB). When I go to download these (as a standalone attachment) the CouchDB process grows in size by at least the size of the attachment before the download starts. This implies that the attachment is being loaded into memory entirely before being sent to the client. Has anyone else seen this behaviour? Is this a bug, or is there a configuration change I can make to resolve this?
>
> I've tried disabling compression on attachments in case it's the compression that's causing the problem.
>
> I'm using 1.0.1.
What does your request look like?
The standalone attachment API does not buffer.
Cheers
Jan
--