URL as document_id

[Chris Anderson <jc...@gmail.com>]

Hello all.

I'm planning to store the results of a web-crawl in CouchDB, and want
to use the page urls as document_ids. I understand that I can get the
same uniq identifier constraints by using an MD5 of the url, but the
raw URL appeals to me.

The only downside to using a URL as the document_id, is that they can
contain a wide set of characters, and can be quite long. It's not
clear from the wiki if there are any practical limitations on
document_ids -- I'm hoping that gives the go-ahead for me to just pour
raw web sewage (URLs) into CouchDB document_ids.

Thanks for any advice/warnings,
Chris

-- 
Chris Anderson
http://jchris.mfdz.com

Re: URL as document_id

[Chris Anderson <jc...@mfdz.com>]

Guby,

I just tried out inserting and requesting documents with ?-mark chars
in the id, and had success using Ruby's CGI.escape on both the PUT and
the GET

here's how it looked.
PUT /my_test_db/http%3A%2F%2Fexample.com%2Ftest%3Fx%3Dfoo%26y%3Dbar

It seems to display fine in the Javascript CouchDB console, as
http://example.com/test?x=foo&y=bar, so I'm pretty certain that the
CGI escaping was handled correctly by CouchDB.

I'm not querying a view here, just fetching a record. I'll try it out
on a view query and see what results.

Chris


On
, Mar 3, 2008 at 3:36 PM, Guby <gu...@gmail.com> wrote:
> Hi Chris
>  When using URLs as IDs you will get in trouble when you have URLs that
>  contain question marks! I don't get it to work anyhow.
>  I think it might be related to the errors I get when using keys, when
>  querying views, that contain question marks or = or &. I actually sent
>  a message about this to the list yesterday and Neil suggested that
>  there might be an error in
>
>  src/CouchDB/mod_couch.erl.
>
>  where this command is called:
>  case regexp:split(RequestUri, "\\?") of
>
>  Hope that helps.
>
>  Best regards
>  G
>
>
>
>
>
>
>  On Mar 3, 2008, at 6:17 PM, Chris Anderson wrote:
>
>  > Hello all.
>  >
>  > I'm planning to store the results of a web-crawl in CouchDB, and want
>  > to use the page urls as document_ids. I understand that I can get the
>  > same uniq identifier constraints by using an MD5 of the url, but the
>  > raw URL appeals to me.
>  >
>  > The only downside to using a URL as the document_id, is that they can
>  > contain a wide set of characters, and can be quite long. It's not
>  > clear from the wiki if there are any practical limitations on
>  > document_ids -- I'm hoping that gives the go-ahead for me to just pour
>  > raw web sewage (URLs) into CouchDB document_ids.
>  >
>  > Thanks for any advice/warnings,
>  > Chris
>  >
>  > --
>  > Chris Anderson
>  > http://jchris.mfdz.com
>
>



-- 
Chris Anderson
http://jchris.mfdz.com

Re: URL as document_id

[Guby <gu...@gmail.com>]

Hi Chris
When using URLs as IDs you will get in trouble when you have URLs that  
contain question marks! I don't get it to work anyhow.
I think it might be related to the errors I get when using keys, when  
querying views, that contain question marks or = or &. I actually sent  
a message about this to the list yesterday and Neil suggested that  
there might be an error in

src/CouchDB/mod_couch.erl.

where this command is called:
case regexp:split(RequestUri, "\\?") of

Hope that helps.

Best regards
G




On Mar 3, 2008, at 6:17 PM, Chris Anderson wrote:

> Hello all.
>
> I'm planning to store the results of a web-crawl in CouchDB, and want
> to use the page urls as document_ids. I understand that I can get the
> same uniq identifier constraints by using an MD5 of the url, but the
> raw URL appeals to me.
>
> The only downside to using a URL as the document_id, is that they can
> contain a wide set of characters, and can be quite long. It's not
> clear from the wiki if there are any practical limitations on
> document_ids -- I'm hoping that gives the go-ahead for me to just pour
> raw web sewage (URLs) into CouchDB document_ids.
>
> Thanks for any advice/warnings,
> Chris
>
> -- 
> Chris Anderson
> http://jchris.mfdz.com