You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Paul Davis <pa...@gmail.com> on 2010/09/24 20:10:09 UTC
Replacing the _external API
At CouchCamp there was a bit of discussion on replacing the _external
API with something a bit more modern to give _external processes more
control over their environment.
The idea was born out of a discussion with Robert Newson who mentioned
that couchdb-lucene really only needs a reverse proxy to put itself in
the same URL namespace. It occurred to us that having a reverse proxy
instead of the current _external stdio protocol would allow lots of
other interesting features like node.js integration, as well as allow
implementors to handle requests in parallel and so on and such forth.
The major drawback that was identified was that if we switched to just
a reverse proxy, people would then be responsible for handling the
process management of their _external handlers. Ie, they'd have to
configure daemon monitoring to make sure the processes stayed up and
what not. The solution we came up with was to include another feature
that did process management. Ie, something that would bring up an OS
process when the server booted, and respawn it if it crashed. There'd
be no connection to the _externals. Other than the basic "just keep a
process up" sort of behaviour, the only other thing I could see adding
is a simple stdio protocol to get configuration values from CouchDB.
Other people have expressed interest in just the process management
functionality as well which makes me think that having the two new
features to replace the _external API would be both easier on
developers as well as providing more functionality.
So now I'm looking for feedback on what other people might think of
this. I'll start working on this fairly soon if I don't hear any major
objections.
HTH,
Paul Davis
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Sep 24, 2010 at 2:26 PM, Norman Barker <no...@gmail.com> wrote:
> Paul,
>
> for us, the process management is critical, we have external process
> code and due to this process code size and age it is possible that it
> may crash (this is not that unusual!), so I am +1 on process
> management and am interested in helping.
>
Yeah, Mikeal's first concern when I mentioned replacing _externals was
having the process management so that we can keep the dead simple
slave process semantics. In talking with him we realized that this
would provide a dead simple way to hook up a node.js handler as well,
so I think the idea has legs.
> As for reverse proxy is this going to be inside couch (well inside
> mochiweb) or a process in front of couch and external processes?
>
> This will add the capability for streaming results from external
> processes so I am +1.
>
The current plan is to make it inside couch using mochiweb and
ibrowse. I know that Benoît had patches land in the recent ibrowse
release for similar reasons so we should be ready to go on those
fronts.
> Norman
>
>
>
> On Fri, Sep 24, 2010 at 12:10 PM, Paul Davis
> <pa...@gmail.com> wrote:
>> At CouchCamp there was a bit of discussion on replacing the _external
>> API with something a bit more modern to give _external processes more
>> control over their environment.
>>
>> The idea was born out of a discussion with Robert Newson who mentioned
>> that couchdb-lucene really only needs a reverse proxy to put itself in
>> the same URL namespace. It occurred to us that having a reverse proxy
>> instead of the current _external stdio protocol would allow lots of
>> other interesting features like node.js integration, as well as allow
>> implementors to handle requests in parallel and so on and such forth.
>>
>> The major drawback that was identified was that if we switched to just
>> a reverse proxy, people would then be responsible for handling the
>> process management of their _external handlers. Ie, they'd have to
>> configure daemon monitoring to make sure the processes stayed up and
>> what not. The solution we came up with was to include another feature
>> that did process management. Ie, something that would bring up an OS
>> process when the server booted, and respawn it if it crashed. There'd
>> be no connection to the _externals. Other than the basic "just keep a
>> process up" sort of behaviour, the only other thing I could see adding
>> is a simple stdio protocol to get configuration values from CouchDB.
>> Other people have expressed interest in just the process management
>> functionality as well which makes me think that having the two new
>> features to replace the _external API would be both easier on
>> developers as well as providing more functionality.
>>
>> So now I'm looking for feedback on what other people might think of
>> this. I'll start working on this fairly soon if I don't hear any major
>> objections.
>>
>> HTH,
>> Paul Davis
>>
>
Re: Replacing the _external API
Posted by Norman Barker <no...@gmail.com>.
Paul,
for us, the process management is critical, we have external process
code and due to this process code size and age it is possible that it
may crash (this is not that unusual!), so I am +1 on process
management and am interested in helping.
As for reverse proxy is this going to be inside couch (well inside
mochiweb) or a process in front of couch and external processes?
This will add the capability for streaming results from external
processes so I am +1.
Norman
On Fri, Sep 24, 2010 at 12:10 PM, Paul Davis
<pa...@gmail.com> wrote:
> At CouchCamp there was a bit of discussion on replacing the _external
> API with something a bit more modern to give _external processes more
> control over their environment.
>
> The idea was born out of a discussion with Robert Newson who mentioned
> that couchdb-lucene really only needs a reverse proxy to put itself in
> the same URL namespace. It occurred to us that having a reverse proxy
> instead of the current _external stdio protocol would allow lots of
> other interesting features like node.js integration, as well as allow
> implementors to handle requests in parallel and so on and such forth.
>
> The major drawback that was identified was that if we switched to just
> a reverse proxy, people would then be responsible for handling the
> process management of their _external handlers. Ie, they'd have to
> configure daemon monitoring to make sure the processes stayed up and
> what not. The solution we came up with was to include another feature
> that did process management. Ie, something that would bring up an OS
> process when the server booted, and respawn it if it crashed. There'd
> be no connection to the _externals. Other than the basic "just keep a
> process up" sort of behaviour, the only other thing I could see adding
> is a simple stdio protocol to get configuration values from CouchDB.
> Other people have expressed interest in just the process management
> functionality as well which makes me think that having the two new
> features to replace the _external API would be both easier on
> developers as well as providing more functionality.
>
> So now I'm looking for feedback on what other people might think of
> this. I'll start working on this fairly soon if I don't hear any major
> objections.
>
> HTH,
> Paul Davis
>
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
"I met a real, live Pick user/admin/developer..."
http://www.chesnok.com/daily/2010/09/11/couchcamp-2010-yay/
On Fri, Sep 24, 2010 at 3:49 PM, Stephen Prater <st...@agrussell.com> wrote:
> PICK story alert!
>
> We have that exact implementation for communication with our PICK servers.
> Works pretty well, and since PICK does
> really horrifying things when you feed it bad data, it provides us another
> level of input sanity checking. (No, we will not accept your ^Y, thank
> you.)
>
> On Sep 24, 2010, at 2:12 PM, Paul Davis wrote:
>
>> Dave,
>>
>> My knee-jerk solution would be to write a small Python script that
>> translates HTTP traffic to the current protocol. Ie, present your
>> scripts with the same interface, but not from inside the Erlang VM.
>>
>> I'm not sure if I'm missing any details that would prevent that from
>> working though.
>>
>> HTH,
>> Paul Davis
>>
>> On Fri, Sep 24, 2010 at 3:03 PM, David Hardtke <da...@cloudant.com> wrote:
>>>
>>> Hi Robert,
>>>
>>> Assume you've got a working program that communicates over stdio and want
>>> to
>>> use it as an external. Or if you're using some antiquated programing
>>> language without good http libraries (octave and matlab, for instance).
>>> stdio is very easy to use.
>>>
>>> Dave
>>>
>>> On 09/24/10 11:46, Robert Newson wrote:
>>>>
>>>> Who would use stdio when a concurrent and HTTP-based alternative was
>>>> available?
>>>>
>>>> On Fri, Sep 24, 2010 at 7:43 PM, David Hardtke<da...@cloudant.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Is it possible to support both? The precedence is query servers, where
>>>>> you
>>>>> can define either a stdio based query_server or a erlang/otp based
>>>>> native_query_server. Depending on what you are trying to do, these
>>>>> multiple options come in handy. In fact, it would be really nice to
>>>>> have
>>>>> 3
>>>>> options for externals -- stdio, reverse proxy, and erlang/otp.
>>>>>
>>>>> Dave
>>>>>
>>>>> On 09/24/10 11:10, Paul Davis wrote:
>>>>>
>>>>>>
>>>>>> At CouchCamp there was a bit of discussion on replacing the _external
>>>>>> API with something a bit more modern to give _external processes more
>>>>>> control over their environment.
>>>>>>
>>>>>> The idea was born out of a discussion with Robert Newson who mentioned
>>>>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>>>>> the same URL namespace. It occurred to us that having a reverse proxy
>>>>>> instead of the current _external stdio protocol would allow lots of
>>>>>> other interesting features like node.js integration, as well as allow
>>>>>> implementors to handle requests in parallel and so on and such forth.
>>>>>>
>>>>>> The major drawback that was identified was that if we switched to just
>>>>>> a reverse proxy, people would then be responsible for handling the
>>>>>> process management of their _external handlers. Ie, they'd have to
>>>>>> configure daemon monitoring to make sure the processes stayed up and
>>>>>> what not. The solution we came up with was to include another feature
>>>>>> that did process management. Ie, something that would bring up an OS
>>>>>> process when the server booted, and respawn it if it crashed. There'd
>>>>>> be no connection to the _externals. Other than the basic "just keep a
>>>>>> process up" sort of behaviour, the only other thing I could see adding
>>>>>> is a simple stdio protocol to get configuration values from CouchDB.
>>>>>> Other people have expressed interest in just the process management
>>>>>> functionality as well which makes me think that having the two new
>>>>>> features to replace the _external API would be both easier on
>>>>>> developers as well as providing more functionality.
>>>>>>
>>>>>> So now I'm looking for feedback on what other people might think of
>>>>>> this. I'll start working on this fairly soon if I don't hear any major
>>>>>> objections.
>>>>>>
>>>>>> HTH,
>>>>>> Paul Davis
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>
>
Re: Replacing the _external API
Posted by Stephen Prater <st...@agrussell.com>.
PICK story alert!
We have that exact implementation for communication with our PICK
servers. Works pretty well, and since PICK does
really horrifying things when you feed it bad data, it provides us
another level of input sanity checking. (No, we will not accept your
^Y, thank you.)
On Sep 24, 2010, at 2:12 PM, Paul Davis wrote:
> Dave,
>
> My knee-jerk solution would be to write a small Python script that
> translates HTTP traffic to the current protocol. Ie, present your
> scripts with the same interface, but not from inside the Erlang VM.
>
> I'm not sure if I'm missing any details that would prevent that from
> working though.
>
> HTH,
> Paul Davis
>
> On Fri, Sep 24, 2010 at 3:03 PM, David Hardtke <da...@cloudant.com>
> wrote:
>> Hi Robert,
>>
>> Assume you've got a working program that communicates over stdio
>> and want to
>> use it as an external. Or if you're using some antiquated programing
>> language without good http libraries (octave and matlab, for
>> instance).
>> stdio is very easy to use.
>>
>> Dave
>>
>> On 09/24/10 11:46, Robert Newson wrote:
>>>
>>> Who would use stdio when a concurrent and HTTP-based alternative was
>>> available?
>>>
>>> On Fri, Sep 24, 2010 at 7:43 PM, David
>>> Hardtke<da...@cloudant.com> wrote:
>>>
>>>>
>>>> Hi Paul,
>>>>
>>>> Is it possible to support both? The precedence is query servers,
>>>> where
>>>> you
>>>> can define either a stdio based query_server or a erlang/otp based
>>>> native_query_server. Depending on what you are trying to do,
>>>> these
>>>> multiple options come in handy. In fact, it would be really nice
>>>> to have
>>>> 3
>>>> options for externals -- stdio, reverse proxy, and erlang/otp.
>>>>
>>>> Dave
>>>>
>>>> On 09/24/10 11:10, Paul Davis wrote:
>>>>
>>>>>
>>>>> At CouchCamp there was a bit of discussion on replacing the
>>>>> _external
>>>>> API with something a bit more modern to give _external processes
>>>>> more
>>>>> control over their environment.
>>>>>
>>>>> The idea was born out of a discussion with Robert Newson who
>>>>> mentioned
>>>>> that couchdb-lucene really only needs a reverse proxy to put
>>>>> itself in
>>>>> the same URL namespace. It occurred to us that having a reverse
>>>>> proxy
>>>>> instead of the current _external stdio protocol would allow lots
>>>>> of
>>>>> other interesting features like node.js integration, as well as
>>>>> allow
>>>>> implementors to handle requests in parallel and so on and such
>>>>> forth.
>>>>>
>>>>> The major drawback that was identified was that if we switched
>>>>> to just
>>>>> a reverse proxy, people would then be responsible for handling the
>>>>> process management of their _external handlers. Ie, they'd have to
>>>>> configure daemon monitoring to make sure the processes stayed up
>>>>> and
>>>>> what not. The solution we came up with was to include another
>>>>> feature
>>>>> that did process management. Ie, something that would bring up
>>>>> an OS
>>>>> process when the server booted, and respawn it if it crashed.
>>>>> There'd
>>>>> be no connection to the _externals. Other than the basic "just
>>>>> keep a
>>>>> process up" sort of behaviour, the only other thing I could see
>>>>> adding
>>>>> is a simple stdio protocol to get configuration values from
>>>>> CouchDB.
>>>>> Other people have expressed interest in just the process
>>>>> management
>>>>> functionality as well which makes me think that having the two new
>>>>> features to replace the _external API would be both easier on
>>>>> developers as well as providing more functionality.
>>>>>
>>>>> So now I'm looking for feedback on what other people might think
>>>>> of
>>>>> this. I'll start working on this fairly soon if I don't hear any
>>>>> major
>>>>> objections.
>>>>>
>>>>> HTH,
>>>>> Paul Davis
>>>>>
>>>>>
>>>>
>>>>
>>
>>
>
>
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
Dave,
My knee-jerk solution would be to write a small Python script that
translates HTTP traffic to the current protocol. Ie, present your
scripts with the same interface, but not from inside the Erlang VM.
I'm not sure if I'm missing any details that would prevent that from
working though.
HTH,
Paul Davis
On Fri, Sep 24, 2010 at 3:03 PM, David Hardtke <da...@cloudant.com> wrote:
> Hi Robert,
>
> Assume you've got a working program that communicates over stdio and want to
> use it as an external. Or if you're using some antiquated programing
> language without good http libraries (octave and matlab, for instance).
> stdio is very easy to use.
>
> Dave
>
> On 09/24/10 11:46, Robert Newson wrote:
>>
>> Who would use stdio when a concurrent and HTTP-based alternative was
>> available?
>>
>> On Fri, Sep 24, 2010 at 7:43 PM, David Hardtke<da...@cloudant.com> wrote:
>>
>>>
>>> Hi Paul,
>>>
>>> Is it possible to support both? The precedence is query servers, where
>>> you
>>> can define either a stdio based query_server or a erlang/otp based
>>> native_query_server. Depending on what you are trying to do, these
>>> multiple options come in handy. In fact, it would be really nice to have
>>> 3
>>> options for externals -- stdio, reverse proxy, and erlang/otp.
>>>
>>> Dave
>>>
>>> On 09/24/10 11:10, Paul Davis wrote:
>>>
>>>>
>>>> At CouchCamp there was a bit of discussion on replacing the _external
>>>> API with something a bit more modern to give _external processes more
>>>> control over their environment.
>>>>
>>>> The idea was born out of a discussion with Robert Newson who mentioned
>>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>>> the same URL namespace. It occurred to us that having a reverse proxy
>>>> instead of the current _external stdio protocol would allow lots of
>>>> other interesting features like node.js integration, as well as allow
>>>> implementors to handle requests in parallel and so on and such forth.
>>>>
>>>> The major drawback that was identified was that if we switched to just
>>>> a reverse proxy, people would then be responsible for handling the
>>>> process management of their _external handlers. Ie, they'd have to
>>>> configure daemon monitoring to make sure the processes stayed up and
>>>> what not. The solution we came up with was to include another feature
>>>> that did process management. Ie, something that would bring up an OS
>>>> process when the server booted, and respawn it if it crashed. There'd
>>>> be no connection to the _externals. Other than the basic "just keep a
>>>> process up" sort of behaviour, the only other thing I could see adding
>>>> is a simple stdio protocol to get configuration values from CouchDB.
>>>> Other people have expressed interest in just the process management
>>>> functionality as well which makes me think that having the two new
>>>> features to replace the _external API would be both easier on
>>>> developers as well as providing more functionality.
>>>>
>>>> So now I'm looking for feedback on what other people might think of
>>>> this. I'll start working on this fairly soon if I don't hear any major
>>>> objections.
>>>>
>>>> HTH,
>>>> Paul Davis
>>>>
>>>>
>>>
>>>
>
>
Re: Replacing the _external API
Posted by David Hardtke <da...@cloudant.com>.
Hi Robert,
Assume you've got a working program that communicates over stdio and
want to use it as an external. Or if you're using some antiquated
programing language without good http libraries (octave and matlab, for
instance). stdio is very easy to use.
Dave
On 09/24/10 11:46, Robert Newson wrote:
> Who would use stdio when a concurrent and HTTP-based alternative was available?
>
> On Fri, Sep 24, 2010 at 7:43 PM, David Hardtke<da...@cloudant.com> wrote:
>
>> Hi Paul,
>>
>> Is it possible to support both? The precedence is query servers, where you
>> can define either a stdio based query_server or a erlang/otp based
>> native_query_server. Depending on what you are trying to do, these
>> multiple options come in handy. In fact, it would be really nice to have 3
>> options for externals -- stdio, reverse proxy, and erlang/otp.
>>
>> Dave
>>
>> On 09/24/10 11:10, Paul Davis wrote:
>>
>>> At CouchCamp there was a bit of discussion on replacing the _external
>>> API with something a bit more modern to give _external processes more
>>> control over their environment.
>>>
>>> The idea was born out of a discussion with Robert Newson who mentioned
>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>> the same URL namespace. It occurred to us that having a reverse proxy
>>> instead of the current _external stdio protocol would allow lots of
>>> other interesting features like node.js integration, as well as allow
>>> implementors to handle requests in parallel and so on and such forth.
>>>
>>> The major drawback that was identified was that if we switched to just
>>> a reverse proxy, people would then be responsible for handling the
>>> process management of their _external handlers. Ie, they'd have to
>>> configure daemon monitoring to make sure the processes stayed up and
>>> what not. The solution we came up with was to include another feature
>>> that did process management. Ie, something that would bring up an OS
>>> process when the server booted, and respawn it if it crashed. There'd
>>> be no connection to the _externals. Other than the basic "just keep a
>>> process up" sort of behaviour, the only other thing I could see adding
>>> is a simple stdio protocol to get configuration values from CouchDB.
>>> Other people have expressed interest in just the process management
>>> functionality as well which makes me think that having the two new
>>> features to replace the _external API would be both easier on
>>> developers as well as providing more functionality.
>>>
>>> So now I'm looking for feedback on what other people might think of
>>> this. I'll start working on this fairly soon if I don't hear any major
>>> objections.
>>>
>>> HTH,
>>> Paul Davis
>>>
>>>
>>
>>
Re: Replacing the _external API
Posted by Robert Newson <ro...@gmail.com>.
Who would use stdio when a concurrent and HTTP-based alternative was available?
On Fri, Sep 24, 2010 at 7:43 PM, David Hardtke <da...@cloudant.com> wrote:
> Hi Paul,
>
> Is it possible to support both? The precedence is query servers, where you
> can define either a stdio based query_server or a erlang/otp based
> native_query_server. Depending on what you are trying to do, these
> multiple options come in handy. In fact, it would be really nice to have 3
> options for externals -- stdio, reverse proxy, and erlang/otp.
>
> Dave
>
> On 09/24/10 11:10, Paul Davis wrote:
>>
>> At CouchCamp there was a bit of discussion on replacing the _external
>> API with something a bit more modern to give _external processes more
>> control over their environment.
>>
>> The idea was born out of a discussion with Robert Newson who mentioned
>> that couchdb-lucene really only needs a reverse proxy to put itself in
>> the same URL namespace. It occurred to us that having a reverse proxy
>> instead of the current _external stdio protocol would allow lots of
>> other interesting features like node.js integration, as well as allow
>> implementors to handle requests in parallel and so on and such forth.
>>
>> The major drawback that was identified was that if we switched to just
>> a reverse proxy, people would then be responsible for handling the
>> process management of their _external handlers. Ie, they'd have to
>> configure daemon monitoring to make sure the processes stayed up and
>> what not. The solution we came up with was to include another feature
>> that did process management. Ie, something that would bring up an OS
>> process when the server booted, and respawn it if it crashed. There'd
>> be no connection to the _externals. Other than the basic "just keep a
>> process up" sort of behaviour, the only other thing I could see adding
>> is a simple stdio protocol to get configuration values from CouchDB.
>> Other people have expressed interest in just the process management
>> functionality as well which makes me think that having the two new
>> features to replace the _external API would be both easier on
>> developers as well as providing more functionality.
>>
>> So now I'm looking for feedback on what other people might think of
>> this. I'll start working on this fairly soon if I don't hear any major
>> objections.
>>
>> HTH,
>> Paul Davis
>>
>
>
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
David,
I think _externals would be deprecated and released after a couple
versions. Is there a specific reason for wanting to keep the stdio
protocol around? As to Erlang externals, I would probably classify
anything running in the same VM as a plugin like what Volker is doing
with GeoCouch.
Also, I don't think that the view server is quite the same metaphor.
The "couch_native_process" is mostly a misnomer that I've apologized
for many times. On the other hand, the idea for "externals" is that
they're always outside the VM, rather than an abstraction for
something inside the VM. If that makes any sense I guess...
HTH,
Paul Davis
On Fri, Sep 24, 2010 at 2:43 PM, David Hardtke <da...@cloudant.com> wrote:
> Hi Paul,
>
> Is it possible to support both? The precedence is query servers, where you
> can define either a stdio based query_server or a erlang/otp based
> native_query_server. Depending on what you are trying to do, these
> multiple options come in handy. In fact, it would be really nice to have 3
> options for externals -- stdio, reverse proxy, and erlang/otp.
>
> Dave
>
> On 09/24/10 11:10, Paul Davis wrote:
>>
>> At CouchCamp there was a bit of discussion on replacing the _external
>> API with something a bit more modern to give _external processes more
>> control over their environment.
>>
>> The idea was born out of a discussion with Robert Newson who mentioned
>> that couchdb-lucene really only needs a reverse proxy to put itself in
>> the same URL namespace. It occurred to us that having a reverse proxy
>> instead of the current _external stdio protocol would allow lots of
>> other interesting features like node.js integration, as well as allow
>> implementors to handle requests in parallel and so on and such forth.
>>
>> The major drawback that was identified was that if we switched to just
>> a reverse proxy, people would then be responsible for handling the
>> process management of their _external handlers. Ie, they'd have to
>> configure daemon monitoring to make sure the processes stayed up and
>> what not. The solution we came up with was to include another feature
>> that did process management. Ie, something that would bring up an OS
>> process when the server booted, and respawn it if it crashed. There'd
>> be no connection to the _externals. Other than the basic "just keep a
>> process up" sort of behaviour, the only other thing I could see adding
>> is a simple stdio protocol to get configuration values from CouchDB.
>> Other people have expressed interest in just the process management
>> functionality as well which makes me think that having the two new
>> features to replace the _external API would be both easier on
>> developers as well as providing more functionality.
>>
>> So now I'm looking for feedback on what other people might think of
>> this. I'll start working on this fairly soon if I don't hear any major
>> objections.
>>
>> HTH,
>> Paul Davis
>>
>
>
Re: Replacing the _external API
Posted by David Hardtke <da...@cloudant.com>.
Hi Paul,
Is it possible to support both? The precedence is query servers, where
you can define either a stdio based query_server or a erlang/otp based
native_query_server. Depending on what you are trying to do, these
multiple options come in handy. In fact, it would be really nice to
have 3 options for externals -- stdio, reverse proxy, and erlang/otp.
Dave
On 09/24/10 11:10, Paul Davis wrote:
> At CouchCamp there was a bit of discussion on replacing the _external
> API with something a bit more modern to give _external processes more
> control over their environment.
>
> The idea was born out of a discussion with Robert Newson who mentioned
> that couchdb-lucene really only needs a reverse proxy to put itself in
> the same URL namespace. It occurred to us that having a reverse proxy
> instead of the current _external stdio protocol would allow lots of
> other interesting features like node.js integration, as well as allow
> implementors to handle requests in parallel and so on and such forth.
>
> The major drawback that was identified was that if we switched to just
> a reverse proxy, people would then be responsible for handling the
> process management of their _external handlers. Ie, they'd have to
> configure daemon monitoring to make sure the processes stayed up and
> what not. The solution we came up with was to include another feature
> that did process management. Ie, something that would bring up an OS
> process when the server booted, and respawn it if it crashed. There'd
> be no connection to the _externals. Other than the basic "just keep a
> process up" sort of behaviour, the only other thing I could see adding
> is a simple stdio protocol to get configuration values from CouchDB.
> Other people have expressed interest in just the process management
> functionality as well which makes me think that having the two new
> features to replace the _external API would be both easier on
> developers as well as providing more functionality.
>
> So now I'm looking for feedback on what other people might think of
> this. I'll start working on this fairly soon if I don't hear any major
> objections.
>
> HTH,
> Paul Davis
>
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
Well, HTTP is implied, but that doesn't require a choice between TCP
or Unix domain sockets. Unless there's something weird in Erlang,
supporting both transports should be fairly trivial.
I'd considered figuring out someway to support arbitrary TCP
connections but the initial complexity level convinced me to ignore
such things till much much later. On the other hand, I'm pretty sure
proxying WebSocket connections should be doable.
HTH,
Paul Davis
On Fri, Sep 24, 2010 at 2:54 PM, Sebastian Cohnen
<se...@googlemail.com> wrote:
> Does your proxy imply http or tcp? I'm thinking about unix sockets, which would be awesome for many kinds of applications.
>
> On 24.09.2010, at 20:10, Paul Davis wrote:
>
>> At CouchCamp there was a bit of discussion on replacing the _external
>> API with something a bit more modern to give _external processes more
>> control over their environment.
>>
>> The idea was born out of a discussion with Robert Newson who mentioned
>> that couchdb-lucene really only needs a reverse proxy to put itself in
>> the same URL namespace. It occurred to us that having a reverse proxy
>> instead of the current _external stdio protocol would allow lots of
>> other interesting features like node.js integration, as well as allow
>> implementors to handle requests in parallel and so on and such forth.
>>
>> The major drawback that was identified was that if we switched to just
>> a reverse proxy, people would then be responsible for handling the
>> process management of their _external handlers. Ie, they'd have to
>> configure daemon monitoring to make sure the processes stayed up and
>> what not. The solution we came up with was to include another feature
>> that did process management. Ie, something that would bring up an OS
>> process when the server booted, and respawn it if it crashed. There'd
>> be no connection to the _externals. Other than the basic "just keep a
>> process up" sort of behaviour, the only other thing I could see adding
>> is a simple stdio protocol to get configuration values from CouchDB.
>> Other people have expressed interest in just the process management
>> functionality as well which makes me think that having the two new
>> features to replace the _external API would be both easier on
>> developers as well as providing more functionality.
>>
>> So now I'm looking for feedback on what other people might think of
>> this. I'll start working on this fairly soon if I don't hear any major
>> objections.
>>
>> HTH,
>> Paul Davis
>
>
Re: Replacing the _external API
Posted by Sebastian Cohnen <se...@googlemail.com>.
Does your proxy imply http or tcp? I'm thinking about unix sockets, which would be awesome for many kinds of applications.
On 24.09.2010, at 20:10, Paul Davis wrote:
> At CouchCamp there was a bit of discussion on replacing the _external
> API with something a bit more modern to give _external processes more
> control over their environment.
>
> The idea was born out of a discussion with Robert Newson who mentioned
> that couchdb-lucene really only needs a reverse proxy to put itself in
> the same URL namespace. It occurred to us that having a reverse proxy
> instead of the current _external stdio protocol would allow lots of
> other interesting features like node.js integration, as well as allow
> implementors to handle requests in parallel and so on and such forth.
>
> The major drawback that was identified was that if we switched to just
> a reverse proxy, people would then be responsible for handling the
> process management of their _external handlers. Ie, they'd have to
> configure daemon monitoring to make sure the processes stayed up and
> what not. The solution we came up with was to include another feature
> that did process management. Ie, something that would bring up an OS
> process when the server booted, and respawn it if it crashed. There'd
> be no connection to the _externals. Other than the basic "just keep a
> process up" sort of behaviour, the only other thing I could see adding
> is a simple stdio protocol to get configuration values from CouchDB.
> Other people have expressed interest in just the process management
> functionality as well which makes me think that having the two new
> features to replace the _external API would be both easier on
> developers as well as providing more functionality.
>
> So now I'm looking for feedback on what other people might think of
> this. I'll start working on this fairly soon if I don't hear any major
> objections.
>
> HTH,
> Paul Davis
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
On Sun, Sep 26, 2010 at 4:59 PM, Paul Davis <pa...@gmail.com> wrote:
> On Sat, Sep 25, 2010 at 7:28 PM, Paul Davis <pa...@gmail.com> wrote:
>> I've implemented the OS daemons feature and added some preliminary etap tests.
>>
>> Patch as commit is here:
>>
>> http://github.com/davisp/couchdb/commit/01b75b8f74cd31605cbdbc3a6f6c7b9e8fcdcbe3
>>
>> This feature affects zero functionality and is purely an addition to
>> CouchDB. I'm tempted to just and commit it because of that but I'm
>> gonna wait another day or two to see if someone comes up with a strong
>> opposition to this line of development.
>>
>> Feed back desired.
>>
>> Paul Davis
>>
>
> Here's the second part of the new API. Its an HTTP proxy. Yep.
>
> http://github.com/davisp/couchdb/tree/http_proxy
>
> Configure it like such:
>
> [httpd_global_handlers]
> _name = {couch_httpd_proxy, handle_proxy_req, <<"http://proxied-domain.tld/">>}
>
> It'll try and be as transparent as possible.
>
> I haven't got any tests for this other than just poking at it with
> cURL because I can't think of a decent way to implement the tests. If
> anyone has a good idea on that, send me a line.
>
> I'm gonna watch a bit more football and then put together a blog post
> that explains the current state of things, the new bits, and then
> shows the new stuff off with something that I think people might find
> interesting.
>
> HTH,
> Paul Davis
>
Here's a blog post that describes the situation with words and a
smidgen of code. Let me know about comments or corrections:
http://davispj.com/2010/09/26/new-couchdb-externals-api.html
Paul Davis
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
On Sat, Sep 25, 2010 at 7:28 PM, Paul Davis <pa...@gmail.com> wrote:
> I've implemented the OS daemons feature and added some preliminary etap tests.
>
> Patch as commit is here:
>
> http://github.com/davisp/couchdb/commit/01b75b8f74cd31605cbdbc3a6f6c7b9e8fcdcbe3
>
> This feature affects zero functionality and is purely an addition to
> CouchDB. I'm tempted to just and commit it because of that but I'm
> gonna wait another day or two to see if someone comes up with a strong
> opposition to this line of development.
>
> Feed back desired.
>
> Paul Davis
>
Here's the second part of the new API. Its an HTTP proxy. Yep.
http://github.com/davisp/couchdb/tree/http_proxy
Configure it like such:
[httpd_global_handlers]
_name = {couch_httpd_proxy, handle_proxy_req, <<"http://proxied-domain.tld/">>}
It'll try and be as transparent as possible.
I haven't got any tests for this other than just poking at it with
cURL because I can't think of a decent way to implement the tests. If
anyone has a good idea on that, send me a line.
I'm gonna watch a bit more football and then put together a blog post
that explains the current state of things, the new bits, and then
shows the new stuff off with something that I think people might find
interesting.
HTH,
Paul Davis
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
On Sun, Sep 26, 2010 at 1:51 AM, Chris Anderson <jc...@apache.org> wrote:
> "inivalid_json at"
> http://github.com/davisp/couchdb/compare/master...os_daemons#L4R128
>
Good catch. Fixed.
> not sure why this is here but it sure is scary:
>
> % Do *NOT* make this file executable.
>
Whoops, had that in too many files. The one file where its supposed to
be, that's what's being tested. Ie, someone forgot to chmod +x their
script before adding it. To make it less scary I've reworded it to:
% Please do not make this file executable as that's the error being tested.
> There are a couple of things missing from the Makefile.am
>
I the rest of the test files to test/etap/Makefile.am. Had you spotted
anything else?
> On the whole, this patch looks great. It'd be cool to see and example
> usage of it aside from the tests, but that belongs in a blog post, not
> the patch.
I'm working on the proxy feature today. Once I get it done I'll be
putting together a blog post to describe both features.
> good work!
>
> Chris
>
Thanks for the feed back.
Paul Davis
Re: Replacing the _external API
Posted by Chris Anderson <jc...@apache.org>.
"inivalid_json at"
http://github.com/davisp/couchdb/compare/master...os_daemons#L4R128
not sure why this is here but it sure is scary:
% Do *NOT* make this file executable.
There are a couple of things missing from the Makefile.am
On the whole, this patch looks great. It'd be cool to see and example
usage of it aside from the tests, but that belongs in a blog post, not
the patch.
good work!
Chris
On Sat, Sep 25, 2010 at 4:58 PM, Paul Davis <pa...@gmail.com> wrote:
> Already found and fixed a couple more bugs. The latest patch will
> always be the last commit on this branch:
>
> http://github.com/davisp/couchdb/tree/os_daemons
>
> On Sat, Sep 25, 2010 at 7:28 PM, Paul Davis <pa...@gmail.com> wrote:
>> I've implemented the OS daemons feature and added some preliminary etap tests.
>>
>> Patch as commit is here:
>>
>> http://github.com/davisp/couchdb/commit/01b75b8f74cd31605cbdbc3a6f6c7b9e8fcdcbe3
>>
>> This feature affects zero functionality and is purely an addition to
>> CouchDB. I'm tempted to just and commit it because of that but I'm
>> gonna wait another day or two to see if someone comes up with a strong
>> opposition to this line of development.
>>
>> Feed back desired.
>>
>> Paul Davis
>>
>>
>> On Fri, Sep 24, 2010 at 6:20 PM, Filipe David Manana
>> <fd...@apache.org> wrote:
>>> Paul, you have my +1
>>>
>>> cheers
>>>
>>> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com> wrote:
>>>> At CouchCamp there was a bit of discussion on replacing the _external
>>>> API with something a bit more modern to give _external processes more
>>>> control over their environment.
>>>>
>>>> The idea was born out of a discussion with Robert Newson who mentioned
>>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>>> the same URL namespace. It occurred to us that having a reverse proxy
>>>> instead of the current _external stdio protocol would allow lots of
>>>> other interesting features like node.js integration, as well as allow
>>>> implementors to handle requests in parallel and so on and such forth.
>>>>
>>>> The major drawback that was identified was that if we switched to just
>>>> a reverse proxy, people would then be responsible for handling the
>>>> process management of their _external handlers. Ie, they'd have to
>>>> configure daemon monitoring to make sure the processes stayed up and
>>>> what not. The solution we came up with was to include another feature
>>>> that did process management. Ie, something that would bring up an OS
>>>> process when the server booted, and respawn it if it crashed. There'd
>>>> be no connection to the _externals. Other than the basic "just keep a
>>>> process up" sort of behaviour, the only other thing I could see adding
>>>> is a simple stdio protocol to get configuration values from CouchDB.
>>>> Other people have expressed interest in just the process management
>>>> functionality as well which makes me think that having the two new
>>>> features to replace the _external API would be both easier on
>>>> developers as well as providing more functionality.
>>>>
>>>> So now I'm looking for feedback on what other people might think of
>>>> this. I'll start working on this fairly soon if I don't hear any major
>>>> objections.
>>>>
>>>> HTH,
>>>> Paul Davis
>>>>
>>>
>>>
>>>
>>> --
>>> Filipe David Manana,
>>> fdmanana@gmail.com, fdmanana@apache.org
>>>
>>> "Reasonable men adapt themselves to the world.
>>> Unreasonable men adapt the world to themselves.
>>> That's why all progress depends on unreasonable men."
>>>
>>
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
On Sun, Sep 26, 2010 at 3:53 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Sun, Sep 26, 2010 at 1:58 AM, Paul Davis <pa...@gmail.com> wrote:
>> Already found and fixed a couple more bugs. The latest patch will
>> always be the last commit on this branch:
>>
>> http://github.com/davisp/couchdb/tree/os_daemons
>>
> Patch looks good for me. Unfortunatly I can't test it until tomorrow.
>
> I'm currently asking myself if it shouldn't be possible also to set
> the number of processes you launch per command. You could eventually
> say: launc 4 processes for this command. So for example someone could
> eventually run a wsgi server behind or any other stuff.
>
> - benoit
>
I contemplated that but decided not to add it. If people ask for it,
I'll just send them a link to the Gunicorn source code.
Paul
Re: Replacing the _external API
Posted by Benoit Chesneau <bc...@gmail.com>.
On Sun, Sep 26, 2010 at 1:58 AM, Paul Davis <pa...@gmail.com> wrote:
> Already found and fixed a couple more bugs. The latest patch will
> always be the last commit on this branch:
>
> http://github.com/davisp/couchdb/tree/os_daemons
>
Patch looks good for me. Unfortunatly I can't test it until tomorrow.
I'm currently asking myself if it shouldn't be possible also to set
the number of processes you launch per command. You could eventually
say: launc 4 processes for this command. So for example someone could
eventually run a wsgi server behind or any other stuff.
- benoit
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
Already found and fixed a couple more bugs. The latest patch will
always be the last commit on this branch:
http://github.com/davisp/couchdb/tree/os_daemons
On Sat, Sep 25, 2010 at 7:28 PM, Paul Davis <pa...@gmail.com> wrote:
> I've implemented the OS daemons feature and added some preliminary etap tests.
>
> Patch as commit is here:
>
> http://github.com/davisp/couchdb/commit/01b75b8f74cd31605cbdbc3a6f6c7b9e8fcdcbe3
>
> This feature affects zero functionality and is purely an addition to
> CouchDB. I'm tempted to just and commit it because of that but I'm
> gonna wait another day or two to see if someone comes up with a strong
> opposition to this line of development.
>
> Feed back desired.
>
> Paul Davis
>
>
> On Fri, Sep 24, 2010 at 6:20 PM, Filipe David Manana
> <fd...@apache.org> wrote:
>> Paul, you have my +1
>>
>> cheers
>>
>> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com> wrote:
>>> At CouchCamp there was a bit of discussion on replacing the _external
>>> API with something a bit more modern to give _external processes more
>>> control over their environment.
>>>
>>> The idea was born out of a discussion with Robert Newson who mentioned
>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>> the same URL namespace. It occurred to us that having a reverse proxy
>>> instead of the current _external stdio protocol would allow lots of
>>> other interesting features like node.js integration, as well as allow
>>> implementors to handle requests in parallel and so on and such forth.
>>>
>>> The major drawback that was identified was that if we switched to just
>>> a reverse proxy, people would then be responsible for handling the
>>> process management of their _external handlers. Ie, they'd have to
>>> configure daemon monitoring to make sure the processes stayed up and
>>> what not. The solution we came up with was to include another feature
>>> that did process management. Ie, something that would bring up an OS
>>> process when the server booted, and respawn it if it crashed. There'd
>>> be no connection to the _externals. Other than the basic "just keep a
>>> process up" sort of behaviour, the only other thing I could see adding
>>> is a simple stdio protocol to get configuration values from CouchDB.
>>> Other people have expressed interest in just the process management
>>> functionality as well which makes me think that having the two new
>>> features to replace the _external API would be both easier on
>>> developers as well as providing more functionality.
>>>
>>> So now I'm looking for feedback on what other people might think of
>>> this. I'll start working on this fairly soon if I don't hear any major
>>> objections.
>>>
>>> HTH,
>>> Paul Davis
>>>
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana@gmail.com, fdmanana@apache.org
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
I've implemented the OS daemons feature and added some preliminary etap tests.
Patch as commit is here:
http://github.com/davisp/couchdb/commit/01b75b8f74cd31605cbdbc3a6f6c7b9e8fcdcbe3
This feature affects zero functionality and is purely an addition to
CouchDB. I'm tempted to just and commit it because of that but I'm
gonna wait another day or two to see if someone comes up with a strong
opposition to this line of development.
Feed back desired.
Paul Davis
On Fri, Sep 24, 2010 at 6:20 PM, Filipe David Manana
<fd...@apache.org> wrote:
> Paul, you have my +1
>
> cheers
>
> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com> wrote:
>> At CouchCamp there was a bit of discussion on replacing the _external
>> API with something a bit more modern to give _external processes more
>> control over their environment.
>>
>> The idea was born out of a discussion with Robert Newson who mentioned
>> that couchdb-lucene really only needs a reverse proxy to put itself in
>> the same URL namespace. It occurred to us that having a reverse proxy
>> instead of the current _external stdio protocol would allow lots of
>> other interesting features like node.js integration, as well as allow
>> implementors to handle requests in parallel and so on and such forth.
>>
>> The major drawback that was identified was that if we switched to just
>> a reverse proxy, people would then be responsible for handling the
>> process management of their _external handlers. Ie, they'd have to
>> configure daemon monitoring to make sure the processes stayed up and
>> what not. The solution we came up with was to include another feature
>> that did process management. Ie, something that would bring up an OS
>> process when the server booted, and respawn it if it crashed. There'd
>> be no connection to the _externals. Other than the basic "just keep a
>> process up" sort of behaviour, the only other thing I could see adding
>> is a simple stdio protocol to get configuration values from CouchDB.
>> Other people have expressed interest in just the process management
>> functionality as well which makes me think that having the two new
>> features to replace the _external API would be both easier on
>> developers as well as providing more functionality.
>>
>> So now I'm looking for feedback on what other people might think of
>> this. I'll start working on this fairly soon if I don't hear any major
>> objections.
>>
>> HTH,
>> Paul Davis
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
Re: Replacing the _external API
Posted by Filipe David Manana <fd...@apache.org>.
Paul, you have my +1
cheers
On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com> wrote:
> At CouchCamp there was a bit of discussion on replacing the _external
> API with something a bit more modern to give _external processes more
> control over their environment.
>
> The idea was born out of a discussion with Robert Newson who mentioned
> that couchdb-lucene really only needs a reverse proxy to put itself in
> the same URL namespace. It occurred to us that having a reverse proxy
> instead of the current _external stdio protocol would allow lots of
> other interesting features like node.js integration, as well as allow
> implementors to handle requests in parallel and so on and such forth.
>
> The major drawback that was identified was that if we switched to just
> a reverse proxy, people would then be responsible for handling the
> process management of their _external handlers. Ie, they'd have to
> configure daemon monitoring to make sure the processes stayed up and
> what not. The solution we came up with was to include another feature
> that did process management. Ie, something that would bring up an OS
> process when the server booted, and respawn it if it crashed. There'd
> be no connection to the _externals. Other than the basic "just keep a
> process up" sort of behaviour, the only other thing I could see adding
> is a simple stdio protocol to get configuration values from CouchDB.
> Other people have expressed interest in just the process management
> functionality as well which makes me think that having the two new
> features to replace the _external API would be both easier on
> developers as well as providing more functionality.
>
> So now I'm looking for feedback on what other people might think of
> this. I'll start working on this fairly soon if I don't hear any major
> objections.
>
> HTH,
> Paul Davis
>
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Sep 24, 2010 at 2:48 PM, Zachary Zolton
<za...@gmail.com> wrote:
> Paul,
>
> Have you given any thought to having external processes being able to
> write to the CouchDB log, or show up in the _active_tasks?
>
For things managed by the process management, I was considering just
merging anything that came from stderr into the CouchDB logs. As to
_active_tasks, that's an interesting idea, if I implement the config
api of stdio, this could be an added fairly easily as well. Though for
things like lucene where Rob talks about having the lucene index on a
different machine, I don't think I'd allow for HTTP based access to
_active_tasks. Or rather, it'd be a lot harder because of how the
_active_tasks internals work.
> For example, it would be neat if I could watch the progress of Lucene
> indexing along side of building view indexes.
>
> That kind of integration makes it more compelling than just using
> other process management and proxy software.
>
The reason for this feature was always to simplify the configuration
overhead so that people can do interesting things without requiring a
full time sysadmin. I could do anything anyone's ever done with
externals using enough proxy fu and support code, it's just more
overhead.
Though I do think the few extra features will be useful regardless.
HTH,
Paul Davis
>
> Cheers,
>
> Zach
>
> On Fri, Sep 24, 2010 at 1:36 PM, Paul Davis <pa...@gmail.com> wrote:
>> On Fri, Sep 24, 2010 at 2:28 PM, Stephen Prater <st...@agrussell.com> wrote:
>>> As one of the people who wanted the external process management, that's a +1
>>> from me (if my vote counts.)
>>>
>>
>> Every vote counts, its how we measure community consensus.
>>
>>> But I like the sound of the reverse proxy protocol for externals too.
>>>
>>> stephen
>>>
>>> On Sep 24, 2010, at 1:19 PM, Robert Newson wrote:
>>>
>>>> Assuming it's straightforward to extend OTP-style process monitoring
>>>> to external processes (and I'm assuming that the couchjs processes are
>>>> so monitored today) then I like the proposal to add both of these
>>>> things.
>>>>
>>>> My obvious motivation is couchdb-lucene so, with that hat on, would
>>>> this mechanism obviate the need for start couchdb-lucene externally
>>>> and make the Python hook script obsolete? I think it does. Finally,
>>>> there are cases where c-l users might wish to locate their c-l server
>>>> on a different box, so we should allow the proxying independently of
>>>> the launch-on-demand-and-keep-me-running bit.
>>>>
>>>> B.
>>>>
>>>> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com>
>>>> wrote:
>>>>>
>>>>> At CouchCamp there was a bit of discussion on replacing the _external
>>>>> API with something a bit more modern to give _external processes more
>>>>> control over their environment.
>>>>>
>>>>> The idea was born out of a discussion with Robert Newson who mentioned
>>>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>>>> the same URL namespace. It occurred to us that having a reverse proxy
>>>>> instead of the current _external stdio protocol would allow lots of
>>>>> other interesting features like node.js integration, as well as allow
>>>>> implementors to handle requests in parallel and so on and such forth.
>>>>>
>>>>> The major drawback that was identified was that if we switched to just
>>>>> a reverse proxy, people would then be responsible for handling the
>>>>> process management of their _external handlers. Ie, they'd have to
>>>>> configure daemon monitoring to make sure the processes stayed up and
>>>>> what not. The solution we came up with was to include another feature
>>>>> that did process management. Ie, something that would bring up an OS
>>>>> process when the server booted, and respawn it if it crashed. There'd
>>>>> be no connection to the _externals. Other than the basic "just keep a
>>>>> process up" sort of behaviour, the only other thing I could see adding
>>>>> is a simple stdio protocol to get configuration values from CouchDB.
>>>>> Other people have expressed interest in just the process management
>>>>> functionality as well which makes me think that having the two new
>>>>> features to replace the _external API would be both easier on
>>>>> developers as well as providing more functionality.
>>>>>
>>>>> So now I'm looking for feedback on what other people might think of
>>>>> this. I'll start working on this fairly soon if I don't hear any major
>>>>> objections.
>>>>>
>>>>> HTH,
>>>>> Paul Davis
>>>>>
>>>>
>>>
>>>
>>
>
Re: Replacing the _external API
Posted by Zachary Zolton <za...@gmail.com>.
Paul,
Have you given any thought to having external processes being able to
write to the CouchDB log, or show up in the _active_tasks?
For example, it would be neat if I could watch the progress of Lucene
indexing along side of building view indexes.
That kind of integration makes it more compelling than just using
other process management and proxy software.
Cheers,
Zach
On Fri, Sep 24, 2010 at 1:36 PM, Paul Davis <pa...@gmail.com> wrote:
> On Fri, Sep 24, 2010 at 2:28 PM, Stephen Prater <st...@agrussell.com> wrote:
>> As one of the people who wanted the external process management, that's a +1
>> from me (if my vote counts.)
>>
>
> Every vote counts, its how we measure community consensus.
>
>> But I like the sound of the reverse proxy protocol for externals too.
>>
>> stephen
>>
>> On Sep 24, 2010, at 1:19 PM, Robert Newson wrote:
>>
>>> Assuming it's straightforward to extend OTP-style process monitoring
>>> to external processes (and I'm assuming that the couchjs processes are
>>> so monitored today) then I like the proposal to add both of these
>>> things.
>>>
>>> My obvious motivation is couchdb-lucene so, with that hat on, would
>>> this mechanism obviate the need for start couchdb-lucene externally
>>> and make the Python hook script obsolete? I think it does. Finally,
>>> there are cases where c-l users might wish to locate their c-l server
>>> on a different box, so we should allow the proxying independently of
>>> the launch-on-demand-and-keep-me-running bit.
>>>
>>> B.
>>>
>>> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com>
>>> wrote:
>>>>
>>>> At CouchCamp there was a bit of discussion on replacing the _external
>>>> API with something a bit more modern to give _external processes more
>>>> control over their environment.
>>>>
>>>> The idea was born out of a discussion with Robert Newson who mentioned
>>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>>> the same URL namespace. It occurred to us that having a reverse proxy
>>>> instead of the current _external stdio protocol would allow lots of
>>>> other interesting features like node.js integration, as well as allow
>>>> implementors to handle requests in parallel and so on and such forth.
>>>>
>>>> The major drawback that was identified was that if we switched to just
>>>> a reverse proxy, people would then be responsible for handling the
>>>> process management of their _external handlers. Ie, they'd have to
>>>> configure daemon monitoring to make sure the processes stayed up and
>>>> what not. The solution we came up with was to include another feature
>>>> that did process management. Ie, something that would bring up an OS
>>>> process when the server booted, and respawn it if it crashed. There'd
>>>> be no connection to the _externals. Other than the basic "just keep a
>>>> process up" sort of behaviour, the only other thing I could see adding
>>>> is a simple stdio protocol to get configuration values from CouchDB.
>>>> Other people have expressed interest in just the process management
>>>> functionality as well which makes me think that having the two new
>>>> features to replace the _external API would be both easier on
>>>> developers as well as providing more functionality.
>>>>
>>>> So now I'm looking for feedback on what other people might think of
>>>> this. I'll start working on this fairly soon if I don't hear any major
>>>> objections.
>>>>
>>>> HTH,
>>>> Paul Davis
>>>>
>>>
>>
>>
>
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Sep 24, 2010 at 2:28 PM, Stephen Prater <st...@agrussell.com> wrote:
> As one of the people who wanted the external process management, that's a +1
> from me (if my vote counts.)
>
Every vote counts, its how we measure community consensus.
> But I like the sound of the reverse proxy protocol for externals too.
>
> stephen
>
> On Sep 24, 2010, at 1:19 PM, Robert Newson wrote:
>
>> Assuming it's straightforward to extend OTP-style process monitoring
>> to external processes (and I'm assuming that the couchjs processes are
>> so monitored today) then I like the proposal to add both of these
>> things.
>>
>> My obvious motivation is couchdb-lucene so, with that hat on, would
>> this mechanism obviate the need for start couchdb-lucene externally
>> and make the Python hook script obsolete? I think it does. Finally,
>> there are cases where c-l users might wish to locate their c-l server
>> on a different box, so we should allow the proxying independently of
>> the launch-on-demand-and-keep-me-running bit.
>>
>> B.
>>
>> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com>
>> wrote:
>>>
>>> At CouchCamp there was a bit of discussion on replacing the _external
>>> API with something a bit more modern to give _external processes more
>>> control over their environment.
>>>
>>> The idea was born out of a discussion with Robert Newson who mentioned
>>> that couchdb-lucene really only needs a reverse proxy to put itself in
>>> the same URL namespace. It occurred to us that having a reverse proxy
>>> instead of the current _external stdio protocol would allow lots of
>>> other interesting features like node.js integration, as well as allow
>>> implementors to handle requests in parallel and so on and such forth.
>>>
>>> The major drawback that was identified was that if we switched to just
>>> a reverse proxy, people would then be responsible for handling the
>>> process management of their _external handlers. Ie, they'd have to
>>> configure daemon monitoring to make sure the processes stayed up and
>>> what not. The solution we came up with was to include another feature
>>> that did process management. Ie, something that would bring up an OS
>>> process when the server booted, and respawn it if it crashed. There'd
>>> be no connection to the _externals. Other than the basic "just keep a
>>> process up" sort of behaviour, the only other thing I could see adding
>>> is a simple stdio protocol to get configuration values from CouchDB.
>>> Other people have expressed interest in just the process management
>>> functionality as well which makes me think that having the two new
>>> features to replace the _external API would be both easier on
>>> developers as well as providing more functionality.
>>>
>>> So now I'm looking for feedback on what other people might think of
>>> this. I'll start working on this fairly soon if I don't hear any major
>>> objections.
>>>
>>> HTH,
>>> Paul Davis
>>>
>>
>
>
Re: Replacing the _external API
Posted by Stephen Prater <st...@agrussell.com>.
As one of the people who wanted the external process management,
that's a +1 from me (if my vote counts.)
But I like the sound of the reverse proxy protocol for externals too.
stephen
On Sep 24, 2010, at 1:19 PM, Robert Newson wrote:
> Assuming it's straightforward to extend OTP-style process monitoring
> to external processes (and I'm assuming that the couchjs processes are
> so monitored today) then I like the proposal to add both of these
> things.
>
> My obvious motivation is couchdb-lucene so, with that hat on, would
> this mechanism obviate the need for start couchdb-lucene externally
> and make the Python hook script obsolete? I think it does. Finally,
> there are cases where c-l users might wish to locate their c-l server
> on a different box, so we should allow the proxying independently of
> the launch-on-demand-and-keep-me-running bit.
>
> B.
>
> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <paul.joseph.davis@gmail.com
> > wrote:
>> At CouchCamp there was a bit of discussion on replacing the _external
>> API with something a bit more modern to give _external processes more
>> control over their environment.
>>
>> The idea was born out of a discussion with Robert Newson who
>> mentioned
>> that couchdb-lucene really only needs a reverse proxy to put itself
>> in
>> the same URL namespace. It occurred to us that having a reverse proxy
>> instead of the current _external stdio protocol would allow lots of
>> other interesting features like node.js integration, as well as allow
>> implementors to handle requests in parallel and so on and such forth.
>>
>> The major drawback that was identified was that if we switched to
>> just
>> a reverse proxy, people would then be responsible for handling the
>> process management of their _external handlers. Ie, they'd have to
>> configure daemon monitoring to make sure the processes stayed up and
>> what not. The solution we came up with was to include another feature
>> that did process management. Ie, something that would bring up an OS
>> process when the server booted, and respawn it if it crashed. There'd
>> be no connection to the _externals. Other than the basic "just keep a
>> process up" sort of behaviour, the only other thing I could see
>> adding
>> is a simple stdio protocol to get configuration values from CouchDB.
>> Other people have expressed interest in just the process management
>> functionality as well which makes me think that having the two new
>> features to replace the _external API would be both easier on
>> developers as well as providing more functionality.
>>
>> So now I'm looking for feedback on what other people might think of
>> this. I'll start working on this fairly soon if I don't hear any
>> major
>> objections.
>>
>> HTH,
>> Paul Davis
>>
>
Re: Replacing the _external API
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Sep 24, 2010 at 2:19 PM, Robert Newson <ro...@gmail.com> wrote:
> Assuming it's straightforward to extend OTP-style process monitoring
> to external processes (and I'm assuming that the couchjs processes are
> so monitored today) then I like the proposal to add both of these
> things.
>
Yeah, intend for the process monitoring to behave almost exactly like
it does current for _external scripts. Just a "restart when dies" sort
of configuration. The config protocol over stdio was a last minute
idea to make it easy for implementors to figure out where their couch
is and what not.
> My obvious motivation is couchdb-lucene so, with that hat on, would
> this mechanism obviate the need for start couchdb-lucene externally
> and make the Python hook script obsolete? I think it does. Finally,
> there are cases where c-l users might wish to locate their c-l server
> on a different box, so we should allow the proxying independently of
> the launch-on-demand-and-keep-me-running bit.
>
> B.
>
They'll be completely separate features for precisely this sort of
reason. I haven't quite figured out the details on proxies to external
hosts. I see the real need, but I feel as though we should have a
method for disabling the external host ability as well.
> On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com> wrote:
>> At CouchCamp there was a bit of discussion on replacing the _external
>> API with something a bit more modern to give _external processes more
>> control over their environment.
>>
>> The idea was born out of a discussion with Robert Newson who mentioned
>> that couchdb-lucene really only needs a reverse proxy to put itself in
>> the same URL namespace. It occurred to us that having a reverse proxy
>> instead of the current _external stdio protocol would allow lots of
>> other interesting features like node.js integration, as well as allow
>> implementors to handle requests in parallel and so on and such forth.
>>
>> The major drawback that was identified was that if we switched to just
>> a reverse proxy, people would then be responsible for handling the
>> process management of their _external handlers. Ie, they'd have to
>> configure daemon monitoring to make sure the processes stayed up and
>> what not. The solution we came up with was to include another feature
>> that did process management. Ie, something that would bring up an OS
>> process when the server booted, and respawn it if it crashed. There'd
>> be no connection to the _externals. Other than the basic "just keep a
>> process up" sort of behaviour, the only other thing I could see adding
>> is a simple stdio protocol to get configuration values from CouchDB.
>> Other people have expressed interest in just the process management
>> functionality as well which makes me think that having the two new
>> features to replace the _external API would be both easier on
>> developers as well as providing more functionality.
>>
>> So now I'm looking for feedback on what other people might think of
>> this. I'll start working on this fairly soon if I don't hear any major
>> objections.
>>
>> HTH,
>> Paul Davis
>>
>
Re: Replacing the _external API
Posted by Robert Newson <ro...@gmail.com>.
Assuming it's straightforward to extend OTP-style process monitoring
to external processes (and I'm assuming that the couchjs processes are
so monitored today) then I like the proposal to add both of these
things.
My obvious motivation is couchdb-lucene so, with that hat on, would
this mechanism obviate the need for start couchdb-lucene externally
and make the Python hook script obsolete? I think it does. Finally,
there are cases where c-l users might wish to locate their c-l server
on a different box, so we should allow the proxying independently of
the launch-on-demand-and-keep-me-running bit.
B.
On Fri, Sep 24, 2010 at 7:10 PM, Paul Davis <pa...@gmail.com> wrote:
> At CouchCamp there was a bit of discussion on replacing the _external
> API with something a bit more modern to give _external processes more
> control over their environment.
>
> The idea was born out of a discussion with Robert Newson who mentioned
> that couchdb-lucene really only needs a reverse proxy to put itself in
> the same URL namespace. It occurred to us that having a reverse proxy
> instead of the current _external stdio protocol would allow lots of
> other interesting features like node.js integration, as well as allow
> implementors to handle requests in parallel and so on and such forth.
>
> The major drawback that was identified was that if we switched to just
> a reverse proxy, people would then be responsible for handling the
> process management of their _external handlers. Ie, they'd have to
> configure daemon monitoring to make sure the processes stayed up and
> what not. The solution we came up with was to include another feature
> that did process management. Ie, something that would bring up an OS
> process when the server booted, and respawn it if it crashed. There'd
> be no connection to the _externals. Other than the basic "just keep a
> process up" sort of behaviour, the only other thing I could see adding
> is a simple stdio protocol to get configuration values from CouchDB.
> Other people have expressed interest in just the process management
> functionality as well which makes me think that having the two new
> features to replace the _external API would be both easier on
> developers as well as providing more functionality.
>
> So now I'm looking for feedback on what other people might think of
> this. I'll start working on this fairly soon if I don't hear any major
> objections.
>
> HTH,
> Paul Davis
>