You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by "Agrawal, Vikas IN BLR SISL" <vi...@siemens.com> on 2010/01/04 13:32:33 UTC
Large documents are not uploading.
Hello All,
I am new to Couch DB. I am trying to upload vdi file (Virtual disk Image file) which is an image file of sun virtual box machine. Size of my file is 3.5 GB. When I try to upload this file it starts to upload and nothing is happening after that i.e. uploading is not successful. I tried with 11 Mb file and it is working fine.
I have also changed the values of max_document_size and max_attachment_chunk_size in defauilt.ini file to 17 GB. But still no progress.
I am using Windows, version 0.10.0 of CouchDB.
With Best Regards,
Vikas Kumar Agrawal
Siemens IT Solutions & Services (SIS)
CTDC-TEC
No # 84, Keonics Electronic City
Hosur Road, Bangalore.
Tel.: +91 80 2511 3422
mailto: vikaskumar.agrawal@siemens.com<ma...@siemens.com>
www.siemens.co.in<http://www.siemens.co.in/>
Re: Large documents are not uploading.
Posted by Prasobh PK <pr...@gmail.com>.
Hi,
How we can upload files in to couchdb using an html file input field...
--
Regards,
Prasobh PK,
Ontash Technologies
Re: Large documents are not uploading.
Posted by Roger Binns <ro...@rogerbinns.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Agrawal, Vikas IN BLR SISL wrote:
> Thanks for your reply. Please let me know how to increase the limit of browser?
When I have done testing in the past with file downloads and uploads with
multi-gigabyte files and 32 bit programs and browsers my usual experience
has been crashes, random truncations, bizarre error messages etc. 64 bit
programs have worked fine.
All that said, are you sure you really want to be storing VDI as attachments
in CouchDB. 17GB attachments aren't exactly the sweet spot. As an
alternative you could store the VDI files on a file server somewhere and
store the filename in CouchDB.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAktC5fQACgkQmOOfHg372QSkvwCgoFPRiPIknNfO26Zs+xkEtz30
cwIAn0q9epWUWrBuaRNnRwd/JikLmTkL
=NvEj
-----END PGP SIGNATURE-----
RE: Large documents are not uploading.
Posted by "Agrawal, Vikas IN BLR SISL" <vi...@siemens.com>.
Hello Nicholas,
Thanks for your reply. Please let me know how to increase the limit of browser?
Regards,
Vikas
-----Original Message-----
From: Nicholas Orr [mailto:nicholas.orr@zxgen.net]
Sent: Tuesday, January 05, 2010 8:25 AM
To: user
Subject: Re: Large documents are not uploading.
Maybe the browser has a limit...
Try curl
2010/1/4 Agrawal, Vikas IN BLR SISL <vi...@siemens.com>
> Hello All,
>
> I am new to Couch DB. I am trying to upload vdi file (Virtual disk Image
> file) which is an image file of sun virtual box machine. Size of my file is
> 3.5 GB. When I try to upload this file it starts to upload and nothing is
> happening after that i.e. uploading is not successful. I tried with 11 Mb
> file and it is working fine.
>
> I have also changed the values of max_document_size and
> max_attachment_chunk_size in defauilt.ini file to 17 GB. But still no
> progress.
>
> I am using Windows, version 0.10.0 of CouchDB.
>
> With Best Regards,
> Vikas Kumar Agrawal
> Siemens IT Solutions & Services (SIS)
> CTDC-TEC
> No # 84, Keonics Electronic City
> Hosur Road, Bangalore.
> Tel.: +91 80 2511 3422
> mailto: vikaskumar.agrawal@siemens.com<mailto:
> vikaskumar.agrawal@siemens.com>
> www.siemens.co.in<http://www.siemens.co.in/>
>
>
Re: Large documents are not uploading.
Posted by Nicholas Orr <ni...@zxgen.net>.
Maybe the browser has a limit...
Try curl
2010/1/4 Agrawal, Vikas IN BLR SISL <vi...@siemens.com>
> Hello All,
>
> I am new to Couch DB. I am trying to upload vdi file (Virtual disk Image
> file) which is an image file of sun virtual box machine. Size of my file is
> 3.5 GB. When I try to upload this file it starts to upload and nothing is
> happening after that i.e. uploading is not successful. I tried with 11 Mb
> file and it is working fine.
>
> I have also changed the values of max_document_size and
> max_attachment_chunk_size in defauilt.ini file to 17 GB. But still no
> progress.
>
> I am using Windows, version 0.10.0 of CouchDB.
>
> With Best Regards,
> Vikas Kumar Agrawal
> Siemens IT Solutions & Services (SIS)
> CTDC-TEC
> No # 84, Keonics Electronic City
> Hosur Road, Bangalore.
> Tel.: +91 80 2511 3422
> mailto: vikaskumar.agrawal@siemens.com<mailto:
> vikaskumar.agrawal@siemens.com>
> www.siemens.co.in<http://www.siemens.co.in/>
>
>
Re: Large documents are not uploading.
Posted by Benoit Chesneau <bc...@gmail.com>.
On Tue, Jan 5, 2010 at 10:54 AM, Agrawal, Vikas IN BLR SISL
<vi...@siemens.com> wrote:
> Hello Benoit,
>
> Thanks for your reply.
> I have 2 GB RAM. What should be the minimum RAM required to upload such a big file?
>
> Regards,
> Vikas
>
It's easy to check, try to upload your file via firefox and watch
memory consumption.
- benoît
RE: Large documents are not uploading.
Posted by Markus Jelsma <ma...@buyways.nl>.
Hello,
Although RAM may limit some actions i truly doubt that its actually RAM
size that limits the max upload size. It would be incredibly foolish for
a browser developer to 1) load all data in RAM and 2) read RAM contents
to socket. A proper, and i believe modern browser developers use this,
technique is streaming from disk to socket, perhaps using a small
buffered reader.
Cheers,
-
Markus Jelsma Buyways B.V.
Technisch Architect Friesestraatweg 215c
http://www.buyways.nl 9743 AD Groningen
Alg. 050-853 6600 KvK 01074105
Tel. 050-853 6620 Fax. 050-3118124
Mob. 06-5025 8350 In: http://www.linkedin.com/in/markus17
On Tue, 2010-01-05 at 15:24 +0530, Agrawal, Vikas IN BLR SISL wrote:
> Hello Benoit,
>
> Thanks for your reply.
> I have 2 GB RAM. What should be the minimum RAM required to upload such a big file?
>
> Regards,
> Vikas
>
> -----Original Message-----
> From: Benoit Chesneau [mailto:bchesneau@gmail.com]
> Sent: Tuesday, January 05, 2010 3:04 PM
> To: user@couchdb.apache.org
> Subject: Re: Large documents are not uploading.
>
> On Mon, Jan 4, 2010 at 1:32 PM, Agrawal, Vikas IN BLR SISL
> <vi...@siemens.com> wrote:
> > Hello All,
> >
> > I am new to Couch DB. I am trying to upload vdi file (Virtual disk Image file) which is an image file of sun virtual box machine. Size of my file is 3.5 GB. When I try to upload this file it starts to upload and nothing is happening after that i.e. uploading is not successful. I tried with 11 Mb file and it is working fine.
> >
> > I have also changed the values of max_document_size and max_attachment_chunk_size in defauilt.ini file to 17 GB. But still no progress.
> >
> > I am using Windows, version 0.10.0 of CouchDB.
> >
>
> I don't think It's possible to upload such size from the browser, you
> will be limited in RAM I guess. Better to use a script. If you use
> couchdbkit you can can stream the sending. But such attachment size
> it's a little big. It may be better to save it on a filesystem or if
> you want it in CouchDB you can split the file in multiple chunks and
> save it in multiple docs, then map them to retrieve final file from
> from CouchDB.
>
> - benoît
RE: Large documents are not uploading.
Posted by "Agrawal, Vikas IN BLR SISL" <vi...@siemens.com>.
Hello Benoit,
Thanks for your reply.
I have 2 GB RAM. What should be the minimum RAM required to upload such a big file?
Regards,
Vikas
-----Original Message-----
From: Benoit Chesneau [mailto:bchesneau@gmail.com]
Sent: Tuesday, January 05, 2010 3:04 PM
To: user@couchdb.apache.org
Subject: Re: Large documents are not uploading.
On Mon, Jan 4, 2010 at 1:32 PM, Agrawal, Vikas IN BLR SISL
<vi...@siemens.com> wrote:
> Hello All,
>
> I am new to Couch DB. I am trying to upload vdi file (Virtual disk Image file) which is an image file of sun virtual box machine. Size of my file is 3.5 GB. When I try to upload this file it starts to upload and nothing is happening after that i.e. uploading is not successful. I tried with 11 Mb file and it is working fine.
>
> I have also changed the values of max_document_size and max_attachment_chunk_size in defauilt.ini file to 17 GB. But still no progress.
>
> I am using Windows, version 0.10.0 of CouchDB.
>
I don't think It's possible to upload such size from the browser, you
will be limited in RAM I guess. Better to use a script. If you use
couchdbkit you can can stream the sending. But such attachment size
it's a little big. It may be better to save it on a filesystem or if
you want it in CouchDB you can split the file in multiple chunks and
save it in multiple docs, then map them to retrieve final file from
from CouchDB.
- benoît
Re: Large documents are not uploading.
Posted by Benoit Chesneau <bc...@gmail.com>.
On Mon, Jan 4, 2010 at 1:32 PM, Agrawal, Vikas IN BLR SISL
<vi...@siemens.com> wrote:
> Hello All,
>
> I am new to Couch DB. I am trying to upload vdi file (Virtual disk Image file) which is an image file of sun virtual box machine. Size of my file is 3.5 GB. When I try to upload this file it starts to upload and nothing is happening after that i.e. uploading is not successful. I tried with 11 Mb file and it is working fine.
>
> I have also changed the values of max_document_size and max_attachment_chunk_size in defauilt.ini file to 17 GB. But still no progress.
>
> I am using Windows, version 0.10.0 of CouchDB.
>
I don't think It's possible to upload such size from the browser, you
will be limited in RAM I guess. Better to use a script. If you use
couchdbkit you can can stream the sending. But such attachment size
it's a little big. It may be better to save it on a filesystem or if
you want it in CouchDB you can split the file in multiple chunks and
save it in multiple docs, then map them to retrieve final file from
from CouchDB.
- benoît