You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Ian Mayo <ia...@planetmayo.com> on 2012/04/25 11:06:19 UTC
Cloaking calls to external API (ElasticSearch)
Hi all,
I'm loving the native REST support in CouchDb - it's robust and intuitive.
As we all know, however, CouchDb doesn't have great search support
(free-text, boolean queries, etc). So, I'm using ElasticSearch [ES]
to provide this capability. It's working really well - ES is robust,
fast, and very versatile.
My problem is that I'd like to avoid having to disparate REST
architectures - running through different hosts. Plus, ES only
provides limited search for URI-based searches - more advanced
searches require a JSON payload like this one to search for a forename
of Jack or Jill:
======
{
"size" : 60,
"query" :
{
"bool" :
{
"should" : [
{ "term" : { "forename" : "Jack" } } ,
{ "term" : { "forename" : "Jill" } }
],
"minimum_number_should_match" : 1
}
}
}
======
So, what I'd like to do is to introduce a capability into my CouchDb
where I express my search terms in the URI, then the custom code
produces a suitable JSON payload, and invisibly redirects the call to
my ElasticSearch server.
So, my client would call
http://host/contactsdb/[something]/search?q=forename:Jack+Jill
My custom JS code would then produce a payload like that above, and
then fire the query to the ES server.
Thus, the API is all based on http://host/contactsdb, and my client
apps do not have to create ES payloads, or know of the presence of ES.
I guess I could put a file into the _attachments folder to do this, so
it would be:
http://host/contactsdb/_design/api/_attachments/search.html?q=forename:Jack+Jill
Is this achievable?
Are there any better ways of doing this?
Cheers,
Ian
Re: Cloaking calls to external API (ElasticSearch)
Posted by Robert Newson <rn...@apache.org>.
Hi Ian,
I found your question perfectly clear (though I may have been the only
one). The way to achieve this is as I described, write a new external
handler and add it to couchdb. For comparison, couchdb-lucene uses a
Python script (https://github.com/rnewson/couchdb-lucene/blob/master/couchdb-external-hook.py)
for this purpose.
When you POST to couchdb at hostname:port/dbname/_fti the body is
passed to the script, and you can then do whatever you like. In my
case, and yours, this is to transform the input into a query to an
external HTTP service. In my case, couchdb-lucene. In yours,
ElasticSearch.
B.
On 25 April 2012 14:14, Ian Mayo <ia...@planetmayo.com> wrote:
> It appears my question could have been more clear. Sorry about that -
> I'm unfamiliar with some of the language/terms used in CouchDb/REST.
>
> I have two servers. One running a CouchDb database. The other runs
> an ElasticServer [ES] database, which automatically indexed the
> CouchDb documents.
>
> ES is queried by collating a JSON search object, and passing that in a
> GET to the search URI.
>
> I'd like to tidy this architecture in two ways:
> 1) Make it appear I'm just using the CouchDb database.
> 2) Allow search query to be expressed purely in a URI
>
> So, I'd like to introduce a capability in CouchDb to handle (redirect)
> the search call - solving 1). I'd also like this capability to be
> able to parse the URI parameters and produce a JSON search object, as
> shown in the initial message.
>
> I have a suspicion that I may be able to provide this capability in an
> _attachment - though I welcome any feedback/correction/clarification.
>
> cheers,
> Ian
Re: Cloaking calls to external API (ElasticSearch)
Posted by Ian Mayo <ia...@planetmayo.com>.
It appears my question could have been more clear. Sorry about that -
I'm unfamiliar with some of the language/terms used in CouchDb/REST.
I have two servers. One running a CouchDb database. The other runs
an ElasticServer [ES] database, which automatically indexed the
CouchDb documents.
ES is queried by collating a JSON search object, and passing that in a
GET to the search URI.
I'd like to tidy this architecture in two ways:
1) Make it appear I'm just using the CouchDb database.
2) Allow search query to be expressed purely in a URI
So, I'd like to introduce a capability in CouchDb to handle (redirect)
the search call - solving 1). I'd also like this capability to be
able to parse the URI parameters and produce a JSON search object, as
shown in the initial message.
I have a suspicion that I may be able to provide this capability in an
_attachment - though I welcome any feedback/correction/clarification.
cheers,
Ian
Re: Cloaking calls to external API (ElasticSearch)
Posted by CGS <cg...@gmail.com>.
Hi Ian,
I don't suppose I understood well your problem, so, one question to get
your point: why do you want to use ES for searching through your CouchDB if
you want a direct search with CouchDB REST API?
If I understood well, you want to start a process (more precisely, ES in
this case) on the server-side using CouchDB, don't you?
Or I got it totally wrong...
CGS
On Wed, Apr 25, 2012 at 11:06 AM, Ian Mayo <ia...@planetmayo.com> wrote:
> Hi all,
> I'm loving the native REST support in CouchDb - it's robust and intuitive.
>
> As we all know, however, CouchDb doesn't have great search support
> (free-text, boolean queries, etc). So, I'm using ElasticSearch [ES]
> to provide this capability. It's working really well - ES is robust,
> fast, and very versatile.
>
> My problem is that I'd like to avoid having to disparate REST
> architectures - running through different hosts. Plus, ES only
> provides limited search for URI-based searches - more advanced
> searches require a JSON payload like this one to search for a forename
> of Jack or Jill:
>
> ======
> {
> "size" : 60,
> "query" :
> {
> "bool" :
> {
> "should" : [
> { "term" : { "forename" : "Jack" } } ,
> { "term" : { "forename" : "Jill" } }
> ],
> "minimum_number_should_match" : 1
> }
> }
> }
> ======
>
> So, what I'd like to do is to introduce a capability into my CouchDb
> where I express my search terms in the URI, then the custom code
> produces a suitable JSON payload, and invisibly redirects the call to
> my ElasticSearch server.
>
> So, my client would call
> http://host/contactsdb/[something]/search?q=forename:Jack+Jill
>
> My custom JS code would then produce a payload like that above, and
> then fire the query to the ES server.
>
> Thus, the API is all based on http://host/contactsdb, and my client
> apps do not have to create ES payloads, or know of the presence of ES.
>
> I guess I could put a file into the _attachments folder to do this, so
> it would be:
>
> http://host/contactsdb/_design/api/_attachments/search.html?q=forename:Jack+Jill
>
> Is this achievable?
>
> Are there any better ways of doing this?
>
> Cheers,
> Ian
>
Re: Cloaking calls to external API (ElasticSearch)
Posted by Robert Newson <rn...@apache.org>.
Hi Ian,
You can certainly add your own entry points to CouchDB's REST API
using "externals". The original version of this is described here:
http://wiki.apache.org/couchdb/ExternalProcesses but I recommend the
much enhanced version described here:
http://davispj.com/2010/09/26/new-couchdb-externals-api.html
B.
On 25 April 2012 11:09, Bob Dionne <di...@dionne-associates.com> wrote:
> You might want to look at couchdb-lucene[1], it's integrated with couchdb.
>
> [1] https://github.com/rnewson/couchdb-lucene
>
> On Apr 25, 2012, at 5:06 AM, Ian Mayo wrote:
>
>> Hi all,
>> I'm loving the native REST support in CouchDb - it's robust and intuitive.
>>
>> As we all know, however, CouchDb doesn't have great search support
>> (free-text, boolean queries, etc). So, I'm using ElasticSearch [ES]
>> to provide this capability. It's working really well - ES is robust,
>> fast, and very versatile.
>>
>> My problem is that I'd like to avoid having to disparate REST
>> architectures - running through different hosts. Plus, ES only
>> provides limited search for URI-based searches - more advanced
>> searches require a JSON payload like this one to search for a forename
>> of Jack or Jill:
>>
>> ======
>> {
>> "size" : 60,
>> "query" :
>> {
>> "bool" :
>> {
>> "should" : [
>> { "term" : { "forename" : "Jack" } } ,
>> { "term" : { "forename" : "Jill" } }
>> ],
>> "minimum_number_should_match" : 1
>> }
>> }
>> }
>> ======
>>
>> So, what I'd like to do is to introduce a capability into my CouchDb
>> where I express my search terms in the URI, then the custom code
>> produces a suitable JSON payload, and invisibly redirects the call to
>> my ElasticSearch server.
>>
>> So, my client would call
>> http://host/contactsdb/[something]/search?q=forename:Jack+Jill
>>
>> My custom JS code would then produce a payload like that above, and
>> then fire the query to the ES server.
>>
>> Thus, the API is all based on http://host/contactsdb, and my client
>> apps do not have to create ES payloads, or know of the presence of ES.
>>
>> I guess I could put a file into the _attachments folder to do this, so
>> it would be:
>> http://host/contactsdb/_design/api/_attachments/search.html?q=forename:Jack+Jill
>>
>> Is this achievable?
>>
>> Are there any better ways of doing this?
>>
>> Cheers,
>> Ian
>
Re: Cloaking calls to external API (ElasticSearch)
Posted by Bob Dionne <di...@dionne-associates.com>.
You might want to look at couchdb-lucene[1], it's integrated with couchdb.
[1] https://github.com/rnewson/couchdb-lucene
On Apr 25, 2012, at 5:06 AM, Ian Mayo wrote:
> Hi all,
> I'm loving the native REST support in CouchDb - it's robust and intuitive.
>
> As we all know, however, CouchDb doesn't have great search support
> (free-text, boolean queries, etc). So, I'm using ElasticSearch [ES]
> to provide this capability. It's working really well - ES is robust,
> fast, and very versatile.
>
> My problem is that I'd like to avoid having to disparate REST
> architectures - running through different hosts. Plus, ES only
> provides limited search for URI-based searches - more advanced
> searches require a JSON payload like this one to search for a forename
> of Jack or Jill:
>
> ======
> {
> "size" : 60,
> "query" :
> {
> "bool" :
> {
> "should" : [
> { "term" : { "forename" : "Jack" } } ,
> { "term" : { "forename" : "Jill" } }
> ],
> "minimum_number_should_match" : 1
> }
> }
> }
> ======
>
> So, what I'd like to do is to introduce a capability into my CouchDb
> where I express my search terms in the URI, then the custom code
> produces a suitable JSON payload, and invisibly redirects the call to
> my ElasticSearch server.
>
> So, my client would call
> http://host/contactsdb/[something]/search?q=forename:Jack+Jill
>
> My custom JS code would then produce a payload like that above, and
> then fire the query to the ES server.
>
> Thus, the API is all based on http://host/contactsdb, and my client
> apps do not have to create ES payloads, or know of the presence of ES.
>
> I guess I could put a file into the _attachments folder to do this, so
> it would be:
> http://host/contactsdb/_design/api/_attachments/search.html?q=forename:Jack+Jill
>
> Is this achievable?
>
> Are there any better ways of doing this?
>
> Cheers,
> Ian