You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Patrick Antivackis <pa...@gmail.com> on 2008/10/15 16:29:07 UTC
Issue : upload an attachment using a PUT method
Hello,
Before I fill a JIRA bug and propose a patch to fix it, i would like to
validate this issue with you.
Issue : Cannot upload a file which size > 64M using PUT method
How to reproduce :
Create a database test
Create a 64M file :
dd count=131072 if=/dev/zero of=dummy64M
upload this file using CURL :
curl -H "Content-Type: application/octet-stream" -T dummy64M
localhost:5984/test/dummy64M/ -v
it works :
[patrick@localhost ~]$ curl -H "Content-Type: application/octet-stream" -T
dummy64M localhost:5984/try/dummy64M/ -v
* About to connect() to localhost port 5984 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 5984 (#0)
> PUT /try/dummy64M/dummy64M HTTP/1.1
> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3 libidn/1.0
> Host: localhost:5984
> Accept: */*
> Content-Type: application/octet-stream
> Content-Length: 67108864
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 64.0M 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0} [data not shown]
100 64.0M 0 0 100 64.0M 0 16.3M 0:00:03 0:00:03 --:--:--
16.3M< HTTP/1.1 201 Created
< Server: CouchDB/0.9.0a-incubating (Erlang OTP/R12B)
< Date: Wed, 15 Oct 2008 14:12:00 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 46
< Cache-Control: must-revalidate
<
{ [data not shown]
100 64.0M 0 46 100 64.0M 9 13.4M 0:00:04 0:00:04 --:--:--
13.0M* Connection #0 to host localhost left intact
* Closing connection #0
{"ok":true,"id":"dummy64M","rev":"1158604681"}
now create a 128M file :
dd count=262144 if=/dev/zero of=dummy128M
upload this file using CURL :
curl -H "Content-Type: application/octet-stream" -T dummy128M
localhost:5984/test/dummy128M/ -v
it breaks :
[patrick@localhost ~]$ curl -H "Content-Type:
application/application/octet-stream" -T dummy128M
localhost:5984/try/dummy128M/ -v
* About to connect() to localhost port 5984 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 5984 (#0)
> PUT /try/dummy128M/dummy128M HTTP/1.1
> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3 libidn/1.0
> Host: localhost:5984
> Accept: */*
> Content-Type: application/application/octet-stream
> Content-Length: 134217728
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 128M 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0} [data not shown]
* select/poll returned error
0 128M 0 0 0 16384 0 2177k 0:01:00 --:--:-- 0:01:00
5333k* Closing connection #0
curl: (55) select/poll returned error
There is no error in couchDB logs.
After debugging the code, I found that the problem occurs in
mochiweb_request line 138 with the gen_tcp:recv call :
recv(Length, Timeout) ->
case gen_tcp:recv(Socket, Length, Timeout) of
{ok, Data} ->
put(?SAVE_RECV, true),
Data;
_ ->
exit(normal)
end.
The answer catched by the case _ is in fact {error, enomem}
There is no error in couchdb as the exit is "normal" !!!
For what i read (see answers from
http://www.google.fr/search?q=16M+gen_tcp%3Arecv), if Length is too big
(some says 16M, for me it's 64M) gen_tcp:recv sends back an enomem error.
I made a patch for this in order to call many time gen_tcp:recv with a 16M
Length, and then concat the binary answers (almost like in the read_chunk
method of the mochiweb_multipart).
Anyway, my questions are :
Is it really a bug or is it a way to force curl to use chunk with a PUT
method (not found any but...)
Should the corrections occurs here or should it be given to mochiweb even if
they seem to use PUT only for small upload files.
Thank you for your answers.
Re: Issue : upload an attachment using a PUT method
Posted by Patrick Antivackis <pa...@gmail.com>.
OK Jan,
in fact it's Transfer-Encoding header.
Seems to work better
Thanks
[patrick@localhost ~]$ curl -H "Content-Type:
application/application/octet-stream" -H "Transfer-Encoding: chunked" -T
dummy128M localhost:5984/try/dummy128M/ -v
* About to connect() to localhost port 5984 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 5984 (#0)
> PUT /try/dummy128M/dummy128M HTTP/1.1
> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3 libidn/1.0
> Host: localhost:5984
> Accept: */*
> Content-Type: application/application/octet-stream
> Transfer-Encoding: chunked
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 128M 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0} [data not shown]
100 128M 0 0 100 128M 0 16.1M 0:00:07 0:00:07 --:--:--
0* Empty reply from server
100 128M 0 0 100 128M 0 16.1M 0:00:07 0:00:07 --:--:--
0* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
* Closing connection #0
Re: Issue : upload an attachment using a PUT method
Posted by Patrick Antivackis <pa...@gmail.com>.
Well, not seems for me :
[patrick@localhost ~]$ curl -H "Content-Type:
application/application/octet-stream" -H "Transfer-Coding: chunked" -T
dummy128M localhost:5984/try/dummy128M/ -v
* About to connect() to localhost port 5984 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 5984 (#0)
> PUT /try/dummy128M/dummy128M HTTP/1.1
> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3 libidn/1.0
> Host: localhost:5984
> Accept: */*
> Content-Type: application/application/octet-stream
> Transfer-Coding: chunked
> Content-Length: 134217728
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 128M 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0} [data not shown]
* select/poll returned error
0 128M 0 0 0 16384 0 2408k 0:00:54 --:--:-- 0:00:54
8000k* Closing connection #0
curl: (55) select/poll returned error
Did you used curl like me with a PUT method or with a POST method ?
Re: Issue : upload an attachment using a PUT method
Posted by Jan Lehnardt <ja...@apache.org>.
On Oct 15, 2008, at 16:29 , Patrick Antivackis wrote:
> Hello,
> Before I fill a JIRA bug and propose a patch to fix it, i would like
> to
> validate this issue with you.
>
> Issue : Cannot upload a file which size > 64M using PUT method
> How to reproduce :
> Create a database test
> Create a 64M file :
> dd count=131072 if=/dev/zero of=dummy64M
>
> upload this file using CURL :
> curl -H "Content-Type: application/octet-stream" -T dummy64M
> localhost:5984/test/dummy64M/ -v
> it works :
>
> [patrick@localhost ~]$ curl -H "Content-Type: application/octet-
> stream" -T
> dummy64M localhost:5984/try/dummy64M/ -v
> * About to connect() to localhost port 5984 (#0)
> * Trying 127.0.0.1... connected
> * Connected to localhost (127.0.0.1) port 5984 (#0)
>> PUT /try/dummy64M/dummy64M HTTP/1.1
>> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/
>> 0.9.8e
> zlib/1.2.3 libidn/1.0
>> Host: localhost:5984
>> Accept: */*
>> Content-Type: application/octet-stream
>> Content-Length: 67108864
>> Expect: 100-continue
>>
> < HTTP/1.1 100 Continue
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 64.0M 0 0 0 0 0 0 --:--:-- --:--:--
> --:--:--
> 0} [data not shown]
> 100 64.0M 0 0 100 64.0M 0 16.3M 0:00:03 0:00:03
> --:--:--
> 16.3M< HTTP/1.1 201 Created
> < Server: CouchDB/0.9.0a-incubating (Erlang OTP/R12B)
> < Date: Wed, 15 Oct 2008 14:12:00 GMT
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 46
> < Cache-Control: must-revalidate
> <
> { [data not shown]
> 100 64.0M 0 46 100 64.0M 9 13.4M 0:00:04 0:00:04
> --:--:--
> 13.0M* Connection #0 to host localhost left intact
>
> * Closing connection #0
> {"ok":true,"id":"dummy64M","rev":"1158604681"}
>
> now create a 128M file :
> dd count=262144 if=/dev/zero of=dummy128M
>
> upload this file using CURL :
> curl -H "Content-Type: application/octet-stream" -T dummy128M
> localhost:5984/test/dummy128M/ -v
> it breaks :
>
> [patrick@localhost ~]$ curl -H "Content-Type:
> application/application/octet-stream" -T dummy128M
> localhost:5984/try/dummy128M/ -v
> * About to connect() to localhost port 5984 (#0)
> * Trying 127.0.0.1... connected
> * Connected to localhost (127.0.0.1) port 5984 (#0)
>> PUT /try/dummy128M/dummy128M HTTP/1.1
>> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/
>> 0.9.8e
> zlib/1.2.3 libidn/1.0
>> Host: localhost:5984
>> Accept: */*
>> Content-Type: application/application/octet-stream
>> Content-Length: 134217728
>> Expect: 100-continue
>>
> < HTTP/1.1 100 Continue
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 128M 0 0 0 0 0 0 --:--:-- --:--:--
> --:--:--
> 0} [data not shown]
> * select/poll returned error
> 0 128M 0 0 0 16384 0 2177k 0:01:00 --:--:--
> 0:01:00
> 5333k* Closing connection #0
>
> curl: (55) select/poll returned error
>
> There is no error in couchDB logs.
>
> After debugging the code, I found that the problem occurs in
> mochiweb_request line 138 with the gen_tcp:recv call :
>
> recv(Length, Timeout) ->
> case gen_tcp:recv(Socket, Length, Timeout) of
> {ok, Data} ->
> put(?SAVE_RECV, true),
> Data;
> _ ->
> exit(normal)
> end.
>
>
> The answer catched by the case _ is in fact {error, enomem}
>
> There is no error in couchdb as the exit is "normal" !!!
>
> For what i read (see answers from
> http://www.google.fr/search?q=16M+gen_tcp%3Arecv), if Length is too
> big
> (some says 16M, for me it's 64M) gen_tcp:recv sends back an enomem
> error.
>
> I made a patch for this in order to call many time gen_tcp:recv with
> a 16M
> Length, and then concat the binary answers (almost like in the
> read_chunk
> method of the mochiweb_multipart).
>
> Anyway, my questions are :
> Is it really a bug or is it a way to force curl to use chunk with a
> PUT
> method (not found any but...)
Set the Transfer-Coding header to "chunked" to force curl to chunk the
request.
I successfully uploaded several gigabytes of data using this.
Not sure about the fix.
Cheers
Jan
--
> Should the corrections occurs here or should it be given to mochiweb
> even if
> they seem to use PUT only for small upload files.
>
> Thank you for your answers.