You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by André Warnier <aw...@ice-sa.com> on 2009/04/17 02:01:51 UTC
Not-even-yet-newbie question
Hi good people on this list.
I was recently at ApacheCON Europe, where I followed the spirited and
spiritual Introduction to CouchDB by J. Chris Anderson and Jan Lehnardt.
I also browsed the CouchDB section on the ASF website. I don't know
Erlang, although I followed the brief tutorial linked to from the
website. It looked simple, which makes me suspect I missed quite a lot.
In fact, I have the impression that I missed a whole lot more than
Erlang, so I thank in advance whowever has the patience to read this and
provide some answers to my questions.
I very much like the "Relax" motto.
What I am still trying to figure out mainly, is if CouchDB would be an
appropriate tool for the following.
We basically manage information and documents for other people, as an
ASP service. We provide various easy ways for companies to upload their
electronic documents of all kinds to a dedicated Internet server; we
then process these documents à la Tikka (but not with Tikka)(extract
meta-data and content), automatically index them, and store on the one
side the meta-data and text content in a search engine à la Lucene (but
not Lucene), and on the other side we store the original electronic
document into a special passive file structure that we developed, and
which has proven capable of storing reliably a few million documents so
far. In that file structure, each document is identified by a unique
"logical number", which we store along with the meta-data in the search
engine. (So far in our case, once a document is stored, it never changes).
Then we provide means for the customer to search and find their
documents through a web interface to the search engine, and to retrieve
the corresponding original documents.
It works well and is very reliable, but slowly we are getting into a
management issue due to the volumes of original electronic documents,
which always increases. That is because our customers never throw away
old documents, and they give us ever more varied data to handle.
So we are concerned about increasing volumes to back up, and even more
about volumes to restore in case something would seriously go wrong.
All the above to indicate that when we ourselves talk about "documents",
we talk about on the one hand a searchable index (which works very well,
takes comparatively very little space and which we do not want to change
for now), and on the other hand, stored corresponding electronic
documents (blobs) identified and accessible via one single "key".
I would be interested to understand if CouchDB would provide a reliable
and efficient replacement for our self-developed and self-maintained
storage structure.
The first question is whether the notion of "document" in CouchDB is
compatible with our own notion of document. I mean, could I define in
CouchDB a document as consisting of a single text "key" (a globally
unique document-id), plus a "blob" of undeterminate size (e.g. a MS-Word
document, or a PDF, or an image, or a CAD drawing, or an email or
whatever). And would I then be able to generate for example a search
result webpage, where next to a document summary I can display a PDF
icon, which when clicked retrieves the corresponding electronic document
from CouchDB and sends it to the browser ?
Another aspect that seems particularly interesting - if I got this right
- is the self-replicating nature of CouchDB, which would allow us to
define say 3 "repositories" located in different places, and which would
automatically synchronise themselves. Yes ?
I also seem to have understood that if one of these repositories
suddenly became unavailable because the big one just hit, a document
request would automatically be satisfied by the next available one in
line. Yes ?
Would there be some way in CouchDB to store one such document, in some
logical group containing the original version (say OpenOffice text),
along with its PDF/A version (which we generate when the document is
originally stored) and with an image of the first page (ditto), in such
a way that by using the "main key" plus some additional parameter, I can
retrieve whichever version I need now ?
Would I need to become proficient in Erlang before I can store a new
document or retrieve a stored one, or can this be done using some simple
call from some interface routine in any programming language ?
(For example, a click on a PDF icon generates a call to a mod_perl
add-on Apache module, which then retrieves the document from CouchDB and
returns it to the browser)(perl can "do JSON" or "do XML" e.g.).
To generalise the above question, for what kind of action would I
necessarily need to know Erlang ?
I'll no doubt have more questions if the answers to the above do not
discourage me, but I promise they will be shorter.
Thanks in advance.
Re: Not-even-yet-newbie question
Posted by Chris Anderson <jc...@apache.org>.
On Thu, Apr 16, 2009 at 5:01 PM, André Warnier <aw...@ice-sa.com> wrote:
> Hi good people on this list.
>
> I was recently at ApacheCON Europe, where I followed the spirited and
> spiritual Introduction to CouchDB by J. Chris Anderson and Jan Lehnardt. I
> also browsed the CouchDB section on the ASF website. I don't know Erlang,
> although I followed the brief tutorial linked to from the website. It
> looked simple, which makes me suspect I missed quite a lot.
Glad you enjoyed the talk!
>
> In fact, I have the impression that I missed a whole lot more than Erlang,
> so I thank in advance whowever has the patience to read this and provide
> some answers to my questions.
>
> I very much like the "Relax" motto.
>
> What I am still trying to figure out mainly, is if CouchDB would be an
> appropriate tool for the following.
>
> We basically manage information and documents for other people, as an ASP
> service. We provide various easy ways for companies to upload their
> electronic documents of all kinds to a dedicated Internet server; we then
> process these documents à la Tikka (but not with Tikka)(extract meta-data
> and content), automatically index them, and store on the one side the
> meta-data and text content in a search engine à la Lucene (but not Lucene),
> and on the other side we store the original electronic document into a
> special passive file structure that we developed, and which has proven
> capable of storing reliably a few million documents so far. In that file
> structure, each document is identified by a unique "logical number", which
> we store along with the meta-data in the search engine. (So far in our
> case, once a document is stored, it never changes).
> Then we provide means for the customer to search and find their documents
> through a web interface to the search engine, and to retrieve the
> corresponding original documents.
>
> It works well and is very reliable, but slowly we are getting into a
> management issue due to the volumes of original electronic documents,
> which always increases. That is because our customers never throw away old
> documents, and they give us ever more varied data to handle.
> So we are concerned about increasing volumes to back up, and even more about
> volumes to restore in case something would seriously go wrong.
>
> All the above to indicate that when we ourselves talk about "documents", we
> talk about on the one hand a searchable index (which works very well, takes
> comparatively very little space and which we do not want to change for now),
> and on the other hand, stored corresponding electronic documents (blobs)
> identified and accessible via one single "key".
>
> I would be interested to understand if CouchDB would provide a reliable and
> efficient replacement for our self-developed and self-maintained storage
> structure.
>
This sounds like pretty much exactly CouchDB's sweet spot.
The interface to CouchDB for 99% of users is not Erlang, but plain old
HTTP. See the Futon Ajax administration application for a powerful
example of using CouchDB's REST API.
> The first question is whether the notion of "document" in CouchDB is
> compatible with our own notion of document. I mean, could I define in
> CouchDB a document as consisting of a single text "key" (a globally unique
> document-id), plus a "blob" of undeterminate size (e.g. a MS-Word document,
> or a PDF, or an image, or a CAD drawing, or an email or whatever). And would
> I then be able to generate for example a search result webpage, where next
> to a document summary I can display a PDF icon, which when clicked retrieves
> the corresponding electronic document from CouchDB and sends it to the
> browser ?
In CouchDB, documents are JSON objects which can consist of arbitrary
key/value pairs as well as deeper nested data.
CouchDB documents also have attachments, which are binary blobs of any
size. Each document can have many attachments. The JSON metadata can
be indexed using CouchDB map reduce views, while attachment data is
not sent to the view server.
For your application you could store each file in CouchDB as an
attachment to a JSON document. The JSON document would hold the
searchable metadata. If you had derived versions (pdf from .doc, etc)
you could also store those as more attachments to the same JSON
document.
The attachments themselves can be served directly from CouchDB. Here's
an example:
http://jchrisa.net/drl/ApacheCon-Talk-Slides/CouchDB-Intro-EU.pdf
This is why I think it's such a great fit for your app.
>
> Another aspect that seems particularly interesting - if I got this right -
> is the self-replicating nature of CouchDB, which would allow us to define
> say 3 "repositories" located in different places, and which would
> automatically synchronise themselves. Yes ?
Almost. CouchDB provides triggered replication, which is the primitive
necessary to accomplish this. Operators are responsible for using
replication as their application sees fit. You could use a cron job to
keep the various databases in sync over time.
>
> I also seem to have understood that if one of these repositories suddenly
> became unavailable because the big one just hit, a document request would
> automatically be satisfied by the next available one in line. Yes ?
You can accomplish this yourself with a simple Apache httpd
configuration directive. CouchDB currently acts as a single node, so
any additional multi-node smarts are up to the operator. We're working
on giving it multi-node smarts, but for now it's up to you. (but
pretty easy)
>
> Would there be some way in CouchDB to store one such document, in some
> logical group containing the original version (say OpenOffice text), along
> with its PDF/A version (which we generate when the document is originally
> stored) and with an image of the first page (ditto), in such a way that by
> using the "main key" plus some additional parameter, I can retrieve
> whichever version I need now ?
See above about derived versions...
>
> Would I need to become proficient in Erlang before I can store a new
> document or retrieve a stored one, or can this be done using some simple
> call from some interface routine in any programming language ?
> (For example, a click on a PDF icon generates a call to a mod_perl add-on
> Apache module, which then retrieves the document from CouchDB and returns it
> to the browser)(perl can "do JSON" or "do XML" e.g.).
>
> To generalise the above question, for what kind of action would I
> necessarily need to know Erlang ?
You'd need Erlang to modify CouchDB itself. 99% of users never see Erlang.
>
> I'll no doubt have more questions if the answers to the above do not
> discourage me, but I promise they will be shorter.
>
> Thanks in advance.
>
No problem, glad you are interested!
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Not-even-yet-newbie question
Posted by Brian Candler <B....@pobox.com>.
On Sun, Apr 19, 2009 at 09:35:40AM -0400, Jan Lehnardt wrote:
>> Does this really mean that if we have, for one customer, 100,000
>> CouchDB "documents" consisting each of minimal meta-data, but with
>> each an attachment that is on average a 3 MB PDF file, that this is
>> all stored in one single 300 GB file ?
>>
>> 1a.
>> If yes, is that not uncomfortable/scary ?
>
> Yes, this is correct.
>
>> (I mean, even nowadays, moving a 300 GB file is not the easiest
>> practical thing to do).
It need some testing. Last time I tried rsync with a 2GB file it didn't
handle it very well, but that was several years ago. Since the file is
append-only (apart from the first 4KB changing) in principle it should be
possible to copy it incrementally fairly easily, writing a small tool to do
so if necessary.
> Yes, if you get this use-case, you might not only want a DB per user but
> a DB per user per day.
However at that point you will lose the ability to have a view which indexes
all the customer's documents. That is, after 365 days you would need to do
365 queries just to locate a document by its metadata. I wouldn't recommend
that.
As others have suggested: you could if you prefer just store a pointer (e.g.
filename) to where the file is stored on some other filesystem. Maybe store
them by SHA1, where the filename is xx/xx/xxxxxxxxxxxxxxxx
As an extension of this idea, you could have 257 couchdb filesystems: one
for the indexes of metadata, and the others each storing 1/256th of the SHA1
space. Probably doesn't buy you much over just using the filesystem natively
though.
Regards,
Brian.
Re: Not-even-yet-newbie question
Posted by Jan Lehnardt <ja...@apache.org>.
On 19 Apr 2009, at 08:51, André Warnier wrote:
> Brian Candler wrote:
> [...]
>
> Brian, thanks extremely for the caveats you provided.
> I'd rather get into this with eyes wide open..
>
> Sorry for asking this, but since I don't yet know who's who here..
>
>> 1. The couchdb database is a single append-only file. Your
>> filesystem needs
>> to support huge files.
> Can someone else comment on the above ?
> Does this really mean that if we have, for one customer, 100,000
> CouchDB "documents" consisting each of minimal meta-data, but with
> each an attachment that is on average a 3 MB PDF file, that this is
> all stored in one single 300 GB file ?
>
> 1a.
> If yes, is that not uncomfortable/scary ?
Yes, this is correct.
> (I mean, even nowadays, moving a 300 GB file is not the easiest
> practical thing to do).
Yes, if you get this use-case, you might not only want a DB per user
but a DB per user per day.
> 2a.
> Is there anything that allows to control this ?
see above.
Cheers
Jan
--
>> 2. Once you get up to terabytes of documents, it may become
>> impractical
>> and/or too slow to compact the database, which involves reading the
>> entire
>> database from start to end and writing a completely new copy.
> Agreed.
>
>> In your case it sounds like you normally just append documents and
>> leave
>> them there forever.
> Not 100%, but overwhelmingly so.
>
> However, suppose you have a customer who leaves, and is
>> no longer paying you for the half terabyte of storage they are using?
> Good practical observation. Due to our excellent service, that does
> not happen very often of course. But we do have the occasional real-
> estate broker or bank among our customers..
> ;-)
>
> Or
>> another who, for legal reasons, requires a document to be purged?
>> (Deleting
>> a document in couchdb just marks it as deleted; it can still be
>> retrieved
>> until a compaction has been done.)
> In fact this rather mimics our current system. Customers tend to
> not remember when they delete a document themselves, and tend to
> accuse the computers of losing it.
>
>> I would suggest that the easiest away round these problems - and
>> also a good
>> way to improve security - is to have a separate couchdb database
>> for each
>> customer.
> That also mimics our current layout.
>
> This still only requires running a single instance of the couchdb
>> server.
> Good to know.
>
> Brian, I really appreciate this information. I would probably have
> found this out as I investigate CouchDB further, but it would have
> taken me a lot more time.
>
>
Re: Not-even-yet-newbie question
Posted by André Warnier <aw...@ice-sa.com>.
Brian Candler wrote:
[...]
Brian, thanks extremely for the caveats you provided.
I'd rather get into this with eyes wide open..
Sorry for asking this, but since I don't yet know who's who here..
>
> 1. The couchdb database is a single append-only file. Your filesystem needs
> to support huge files.
>
Can someone else comment on the above ?
Does this really mean that if we have, for one customer, 100,000 CouchDB
"documents" consisting each of minimal meta-data, but with each an
attachment that is on average a 3 MB PDF file, that this is all stored
in one single 300 GB file ?
1a.
If yes, is that not uncomfortable/scary ?
(I mean, even nowadays, moving a 300 GB file is not the easiest
practical thing to do).
2a.
Is there anything that allows to control this ?
> 2. Once you get up to terabytes of documents, it may become impractical
> and/or too slow to compact the database, which involves reading the entire
> database from start to end and writing a completely new copy.
>
Agreed.
> In your case it sounds like you normally just append documents and leave
> them there forever.
Not 100%, but overwhelmingly so.
However, suppose you have a customer who leaves, and is
> no longer paying you for the half terabyte of storage they are using?
Good practical observation. Due to our excellent service, that does not
happen very often of course. But we do have the occasional real-estate
broker or bank among our customers..
;-)
Or
> another who, for legal reasons, requires a document to be purged? (Deleting
> a document in couchdb just marks it as deleted; it can still be retrieved
> until a compaction has been done.)
In fact this rather mimics our current system. Customers tend to not
remember when they delete a document themselves, and tend to accuse the
computers of losing it.
> I would suggest that the easiest away round these problems - and also a good
> way to improve security - is to have a separate couchdb database for each
> customer.
That also mimics our current layout.
This still only requires running a single instance of the couchdb
> server.
Good to know.
Brian, I really appreciate this information. I would probably have
found this out as I investigate CouchDB further, but it would have taken
me a lot more time.
Re: Not-even-yet-newbie question
Posted by Brian Candler <B....@pobox.com>.
On Fri, Apr 17, 2009 at 02:01:51AM +0200, André Warnier wrote:
> I would be interested to understand if CouchDB would provide a reliable
> and efficient replacement for our self-developed and self-maintained
> storage structure.
Probably, but there are a few caveats to beware of.
1. The couchdb database is a single append-only file. Your filesystem needs
to support huge files.
2. Once you get up to terabytes of documents, it may become impractical
and/or too slow to compact the database, which involves reading the entire
database from start to end and writing a completely new copy.
In your case it sounds like you normally just append documents and leave
them there forever. However, suppose you have a customer who leaves, and is
no longer paying you for the half terabyte of storage they are using? Or
another who, for legal reasons, requires a document to be purged? (Deleting
a document in couchdb just marks it as deleted; it can still be retrieved
until a compaction has been done.)
I would suggest that the easiest away round these problems - and also a good
way to improve security - is to have a separate couchdb database for each
customer. This still only requires running a single instance of the couchdb
server.
> I also seem to have understood that if one of these repositories
> suddenly became unavailable because the big one just hit, a document
> request would automatically be satisfied by the next available one in
> line. Yes ?
Not as yet. It's up to you to proxy to the appropriate database in your
application.
> Would there be some way in CouchDB to store one such document, in some
> logical group containing the original version (say OpenOffice text),
> along with its PDF/A version (which we generate when the document is
> originally stored) and with an image of the first page (ditto), in such
> a way that by using the "main key" plus some additional parameter, I can
> retrieve whichever version I need now ?
Document plus multiple attachments. The "document" would just be the
searchable metadata, whilst the original and derived versions would each be
attachments.
> Would I need to become proficient in Erlang before I can store a new
> document or retrieve a stored one
Nope: just HTTP PUT and GET.
Regards,
Brian.
RE: Not-even-yet-newbie question
Posted by Boyd Ebsworthy <be...@gmail.com>.
Hello,
As much as I believe couchdb is a great tool my opinion would be that it is not the right one for that job.
Couch would probably be a perfect tool to store and search the metadata of your documents but for the document itself I believe it wouldn't be practical. I'm still a newbie but as other pointed out couchdb database are only one file which would grow very quickly (millions documents) and the feature of couchdb (JSON documents, map-reduce view, etc..) would not even be used since you only have a single key<=>document association.
You could as some suggested build your own database "sharding" layer but it really seem to be a tough fight for little gain.
What you describe sounds more like you are looking for a distributed fault tolerant parallel filesystem akin to what google, flickr, amazon have designed for themselves. I have seen a presentation somewhere of flickr's filesystem design and it really seemed to fit (lots of doc, few deletions, etc..) but I can't find it anymore.(sorry)
If I were you I'd have a look at Hadoop DFS (another apache project), I don't know how good it is but it seems closer to what you're looking for.
Ref:
http://hadoop.apache.org/core/docs/current/hdfs_design.html
Cheers,
Boyd
-----Original Message-----
From: André Warnier [mailto:aw@ice-sa.com]
Sent: Friday, April 17, 2009 04:02
To: user@couchdb.apache.org
Subject: Not-even-yet-newbie question
Hi good people on this list.
I was recently at ApacheCON Europe, where I followed the spirited and
spiritual Introduction to CouchDB by J. Chris Anderson and Jan Lehnardt.
I also browsed the CouchDB section on the ASF website. I don't know
Erlang, although I followed the brief tutorial linked to from the
website. It looked simple, which makes me suspect I missed quite a lot.
In fact, I have the impression that I missed a whole lot more than
Erlang, so I thank in advance whowever has the patience to read this and
provide some answers to my questions.
I very much like the "Relax" motto.
What I am still trying to figure out mainly, is if CouchDB would be an
appropriate tool for the following.
We basically manage information and documents for other people, as an
ASP service. We provide various easy ways for companies to upload their
electronic documents of all kinds to a dedicated Internet server; we
then process these documents à la Tikka (but not with Tikka)(extract
meta-data and content), automatically index them, and store on the one
side the meta-data and text content in a search engine à la Lucene (but
not Lucene), and on the other side we store the original electronic
document into a special passive file structure that we developed, and
which has proven capable of storing reliably a few million documents so
far. In that file structure, each document is identified by a unique
"logical number", which we store along with the meta-data in the search
engine. (So far in our case, once a document is stored, it never changes).
Then we provide means for the customer to search and find their
documents through a web interface to the search engine, and to retrieve
the corresponding original documents.
It works well and is very reliable, but slowly we are getting into a
management issue due to the volumes of original electronic documents,
which always increases. That is because our customers never throw away
old documents, and they give us ever more varied data to handle.
So we are concerned about increasing volumes to back up, and even more
about volumes to restore in case something would seriously go wrong.
All the above to indicate that when we ourselves talk about "documents",
we talk about on the one hand a searchable index (which works very well,
takes comparatively very little space and which we do not want to change
for now), and on the other hand, stored corresponding electronic
documents (blobs) identified and accessible via one single "key".
I would be interested to understand if CouchDB would provide a reliable
and efficient replacement for our self-developed and self-maintained
storage structure.
The first question is whether the notion of "document" in CouchDB is
compatible with our own notion of document. I mean, could I define in
CouchDB a document as consisting of a single text "key" (a globally
unique document-id), plus a "blob" of undeterminate size (e.g. a MS-Word
document, or a PDF, or an image, or a CAD drawing, or an email or
whatever). And would I then be able to generate for example a search
result webpage, where next to a document summary I can display a PDF
icon, which when clicked retrieves the corresponding electronic document
from CouchDB and sends it to the browser ?
Another aspect that seems particularly interesting - if I got this right
- is the self-replicating nature of CouchDB, which would allow us to
define say 3 "repositories" located in different places, and which would
automatically synchronise themselves. Yes ?
I also seem to have understood that if one of these repositories
suddenly became unavailable because the big one just hit, a document
request would automatically be satisfied by the next available one in
line. Yes ?
Would there be some way in CouchDB to store one such document, in some
logical group containing the original version (say OpenOffice text),
along with its PDF/A version (which we generate when the document is
originally stored) and with an image of the first page (ditto), in such
a way that by using the "main key" plus some additional parameter, I can
retrieve whichever version I need now ?
Would I need to become proficient in Erlang before I can store a new
document or retrieve a stored one, or can this be done using some simple
call from some interface routine in any programming language ?
(For example, a click on a PDF icon generates a call to a mod_perl
add-on Apache module, which then retrieves the document from CouchDB and
returns it to the browser)(perl can "do JSON" or "do XML" e.g.).
To generalise the above question, for what kind of action would I
necessarily need to know Erlang ?
I'll no doubt have more questions if the answers to the above do not
discourage me, but I promise they will be shorter.
Thanks in advance.