You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Bob Clary <bo...@bclary.com> on 2010/03/09 13:28:58 UTC
Problems with large inline attachments with 0.11
Hey all!
I've been using 0.10.1 to successfully post large inline plain text
attachments to couchdb. Now that 0.11 supports automatic compression of
attachments, I thought I would give it a whirl.
I have two issues that I can't find in jira.
1. The cpu and memory usage is extreme for attachments in the 1M to 10M
range. A 64bit linux vm with dual 3Ghz cpu and 3.5G will routinely peg
when updating a document.
2. The server fails to process larger attachments. The server pegs the
cpu and ram and ultimately refuses the connection.
Are these known issues?
Are inline attachments inappropriate here?
I am using http://svn.apache.org/repos/asf/couchdb/trunk Revision:
920758 on Centos5 64bit
Thanks,
Bob Clary
Re: Re: Problems with large inline attachments with 0.11
Posted by Bob Clary <bo...@bclary.com>.
On 7/22/64 11:59 AM, Paul Davis wrote:
> Bob,
>
> Do you see the same issues when using the standalone attachments API?
Paul,
No. Using a simple curl -T -H "Content-Type: text/plain"
http://example:5984/db/doc/file.txt with a 48M text file is handled with
ease.
Re: Problems with large inline attachments with 0.11
Posted by Paul Davis <pa...@gmail.com>.
Bob,
Do you see the same issues when using the standalone attachments API?
Paul
On Tue, Mar 9, 2010 at 7:28 AM, Bob Clary <bo...@bclary.com> wrote:
> Hey all!
>
> I've been using 0.10.1 to successfully post large inline plain text
> attachments to couchdb. Now that 0.11 supports automatic compression of
> attachments, I thought I would give it a whirl.
>
> I have two issues that I can't find in jira.
>
> 1. The cpu and memory usage is extreme for attachments in the 1M to 10M
> range. A 64bit linux vm with dual 3Ghz cpu and 3.5G will routinely peg when
> updating a document.
>
> 2. The server fails to process larger attachments. The server pegs the cpu
> and ram and ultimately refuses the connection.
>
> Are these known issues?
>
> Are inline attachments inappropriate here?
>
> I am using http://svn.apache.org/repos/asf/couchdb/trunk Revision: 920758
> on Centos5 64bit
>
> Thanks,
> Bob Clary
>
Re: Problems with large inline attachments with 0.11
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Mar 12, 2010 at 7:14 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Fri, Mar 12, 2010 at 11:47 PM, Paul Davis
> <pa...@gmail.com> wrote:
>>
>> Isn't there a multipart-mime PUT thing now? I don't think the "put
>> attachment without changing the _rev" behavior is a good idea.
>>
>> Paul
>>
> After thinking about it keeping the _rev behavior is better to handle
> conflicts in replication. If there is a method to send multiple
> attachments in a streaming fashion, it would be obviously better.
>
> - benoît
>
It does break mvcc. Though that multipart-mime action should fix your
"upload all this stuff as one revision" issue.
Paul
Re: Problems with large inline attachments with 0.11
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, Mar 12, 2010 at 11:47 PM, Paul Davis
<pa...@gmail.com> wrote:
>
> Isn't there a multipart-mime PUT thing now? I don't think the "put
> attachment without changing the _rev" behavior is a good idea.
>
> Paul
>
After thinking about it keeping the _rev behavior is better to handle
conflicts in replication. If there is a method to send multiple
attachments in a streaming fashion, it would be obviously better.
- benoît
Re: Problems with large inline attachments with 0.11
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Mar 12, 2010 at 5:26 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Fri, Mar 12, 2010 at 9:14 PM, Adam Kocoloski <ko...@apache.org> wrote:
>> Regardless of the efficiency of term_to_binary, there's really no comparison between the standalone attachment API streaming bytes almost directly to disk and the inline one buffering the whole attachment in memory and inflating it by 37% to boot. Definitely use the standalone API if at all possible. Best,
>>
>> Adam
>>
>
> About that, it would be good to have an option to attach to a specific
> revision of a doc without incrementing it. I know that's the reason
> why couchapp use inline attachments. Some complained it increases the
> number of revisions when you send a lot of attachments.
>
> What do you think ?
>
> - benoit
>
Isn't there a multipart-mime PUT thing now? I don't think the "put
attachment without changing the _rev" behavior is a good idea.
Paul
Re: Problems with large inline attachments with 0.11
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, Mar 12, 2010 at 9:14 PM, Adam Kocoloski <ko...@apache.org> wrote:
> Regardless of the efficiency of term_to_binary, there's really no comparison between the standalone attachment API streaming bytes almost directly to disk and the inline one buffering the whole attachment in memory and inflating it by 37% to boot. Definitely use the standalone API if at all possible. Best,
>
> Adam
>
About that, it would be good to have an option to attach to a specific
revision of a doc without incrementing it. I know that's the reason
why couchapp use inline attachments. Some complained it increases the
number of revisions when you send a lot of attachments.
What do you think ?
- benoit
Re: Problems with large inline attachments with 0.11
Posted by Adam Kocoloski <ko...@apache.org>.
Regardless of the efficiency of term_to_binary, there's really no comparison between the standalone attachment API streaming bytes almost directly to disk and the inline one buffering the whole attachment in memory and inflating it by 37% to boot. Definitely use the standalone API if at all possible. Best,
Adam
On Mar 11, 2010, at 2:07 PM, Randall Leeds wrote:
> I think that's before the fix. I don't know for sure if it will help, but if
> you're up to the task try compiling the latest erlang from source and see if
> it helps.
>
> On Mar 11, 2010 4:33 AM, "Bob Clary" <bo...@bclary.com> wrote:
>
> On 3/10/10 10:24 PM, Randall Leeds wrote:
>>
>> Could you please check your erlang version?
>> I'm wond...
> Name : erlang
> Arch : x86_64
> Version : R12B
> Release : 5.6.el5.1
Re: Problems with large inline attachments with 0.11
Posted by Randall Leeds <ra...@gmail.com>.
I think that's before the fix. I don't know for sure if it will help, but if
you're up to the task try compiling the latest erlang from source and see if
it helps.
On Mar 11, 2010 4:33 AM, "Bob Clary" <bo...@bclary.com> wrote:
On 3/10/10 10:24 PM, Randall Leeds wrote:
>
> Could you please check your erlang version?
> I'm wond...
Name : erlang
Arch : x86_64
Version : R12B
Release : 5.6.el5.1
Re: Problems with large inline attachments with 0.11
Posted by Bob Clary <bo...@bclary.com>.
On 3/10/10 10:24 PM, Randall Leeds wrote:
> Could you please check your erlang version?
> I'm wondering if this is an older, poor implementation of term_to_binary at
> fault. I bet it's called somewhere in there. If you're on R13B you should be
> fine, but I forget exact versions.
Name : erlang
Arch : x86_64
Version : R12B
Release : 5.6.el5.1
Re: Problems with large inline attachments with 0.11
Posted by Randall Leeds <ra...@gmail.com>.
Could you please check your erlang version?
I'm wondering if this is an older, poor implementation of term_to_binary at
fault. I bet it's called somewhere in there. If you're on R13B you should be
fine, but I forget exact versions.
On Mar 10, 2010 10:19 PM, "Jan Lehnardt" <ja...@apache.org> wrote:
On 9 Mar 2010, at 04:28, Bob Clary wrote:
> Hey all!
>
> I've been using 0.10.1 to successfully p...
I'd prefer the standalone attachment API in ay case. The inline attachments
are really just a simplicity stop-gap for the replicator that now supports
the standalone API as well.
Cheers
Jan
--
>
> I am using http://svn.apache.org/repos/asf/couchdb/trunk Revision: 920758
on Centos5 64bit
>...
Re: Problems with large inline attachments with 0.11
Posted by Jan Lehnardt <ja...@apache.org>.
On 9 Mar 2010, at 04:28, Bob Clary wrote:
> Hey all!
>
> I've been using 0.10.1 to successfully post large inline plain text attachments to couchdb. Now that 0.11 supports automatic compression of attachments, I thought I would give it a whirl.
>
> I have two issues that I can't find in jira.
>
> 1. The cpu and memory usage is extreme for attachments in the 1M to 10M range. A 64bit linux vm with dual 3Ghz cpu and 3.5G will routinely peg when updating a document.
>
> 2. The server fails to process larger attachments. The server pegs the cpu and ram and ultimately refuses the connection.
>
> Are these known issues?
>
> Are inline attachments inappropriate here?
I'd prefer the standalone attachment API in ay case. The inline attachments are really just a simplicity stop-gap for the replicator that now supports the standalone API as well.
Cheers
Jan
--
>
> I am using http://svn.apache.org/repos/asf/couchdb/trunk Revision: 920758 on Centos5 64bit
>
> Thanks,
> Bob Clary