os_daemons and query_servers

[Norman Barker <no...@gmail.com>]

Hi,

is there a way to combine query_servers and os_daemons, currently I have

[query_servers]
lang = path/to/interpreter args

and I would like

[os_daemons]
lang_daemon = path/to/interpreter args

[query_servers]
lang = lang_daemon

so as to take advantage of the os_daemons code refactor.

Is this possible?

thanks,

Norman

Re: os_daemons and query_servers

[Paul Davis <pa...@gmail.com>]

> os_daemons have limited ability to request information from couch and
> they initiate.
> "Restart me if configuration section <section> changes" is one (the
> only?) example.

There are three types of requests an os_daemon can make: get config
variable, register to be restarted on config change, or log message.

The formats for each are:

Get config:

    Req: ["get", Section]
    Res: {"key1": "value", "key2": "value2"}

    Req: ["get", Section, Key]
    Res: "value"

Register for restart:

    Req: ["register", Section]
    # process is killed when Section changes.

    Req: ["register", Section, Key]
    # process is killed when Section.Key changes

Log

    Req: ["log", Message]
    # Message is logged at INFO level

    Req: ["log", Message, Opts]
    # Opts current supports a level key like:
    # {"level": "error"} and then we log at that level
    # log message is logged at specified level, or
    # at info level if the level is invalid.

Re: os_daemons and query_servers

[Randall Leeds <ra...@gmail.com>]

On Wed, Jan 26, 2011 at 16:03, Norman Barker <no...@gmail.com> wrote:
> Paul,
>
> understand the answer is no (currently), then why?
>
> a view server can be an external process, python for example, so now
> we have two external managers within couch, why? Why fix a bug in two
> places (I am verifying a timeout bug with the query_servers for an
> external process).

The essential difference between the two is the direction of
request/response flow.
With a synchronous line protocol this communication must be initiated
by one side or the other.

With query servers, couch initiates:
"Prepare these functions", "Map this document" and "Perform this
reduce" are examples of requests couch will send.

os_daemons have limited ability to request information from couch and
they initiate.
"Restart me if configuration section <section> changes" is one (the
only?) example.

Basically, couch doesn't care what the os_daemon process does. If an
os_daemon wants to communicate with couch it should use the HTTP
interface. Paul's example of an updater notifier would do this by
requesting /_changes like any other client.

>
> Is combining os_daemons with external query_servers worth doing, if I
> do this and add a patch can I get this into the trunk?
>

The protocol would have to change to support combining the two
completely. Since the query server API must be preserved between minor
revisions, this could hit 2.0 at the earliest.

Rather than combine the two, perhaps look for patterns/utility
functions that are or could be common to both and abstract that into a
third module. I would be +1 on any patch which does this without
changing the behavior of either.

Randall

Re: os_daemons and query_servers

[Norman Barker <no...@gmail.com>]

Paul,

thanks for the os_daemon code, it is great.

I found the timeout bug we were seeing, it is in
couch_query_servers:get_os_process and
couch_query_servers:get_ddoc_process

there is a gen_server:call that uses the default of 5 seconds, if the
external process takes longer than  5 seconds to start (which in our
case it sometimes does) then we get a bunch of timeout errors, I used
gen_server:call with infinity and it works fine.

Thanks for the help,

Norman



On Wed, Jan 26, 2011 at 5:52 PM, Paul Davis <pa...@gmail.com> wrote:
> On Wed, Jan 26, 2011 at 7:03 PM, Norman Barker <no...@gmail.com> wrote:
>> Paul,
>>
>> understand the answer is no (currently), then why?
>>
>
> As Randall says, the protocol between the two types of OS processes is
> quite different. Each protocol is generally a request/response type of
> protocol, except the actors are switched. In the view server code, it
> is Erlang making requests to the OS process, and for os_daemons, it is
> the OS process that makes the request. Interleaving the two would
> undoubtedly cause more headaches for both the Erlang implementation
> and anyone wanting to write an OS process that implemented one or both
> protocols.
>
> That said, there's not really a whole lot of code in regards to os
> daemons. I think you'd be hard pressed to find examples of where we
> could share code between the two processes without overgeneralizing
> functions to the point where they're harder to reason about. The
> entire os_daemons module [1] (which is almost entirely self contained)
> weighs in at 364 lines (total, not discounting blank lines or
> whitespace).
>
>> a view server can be an external process, python for example, so now
>> we have two external managers within couch, why? Why fix a bug in two
>> places (I am verifying a timeout bug with the query_servers for an
>> external process).
>
> I will give you five internets if you show me a timeout bug that
> exists in both the os_daemons code and view_server code.
>
> That said, the os_daemons code was written after the initial view
> server implementation using a couple paradigms that I've learned since
> last hacking on the view managers. There are definitely bits we could
> port back to the view server manager to simplify a lot of logic.
>
>> Is combining os_daemons with external query_servers worth doing, if I
>> do this and add a patch can I get this into the trunk?
>>
>
> If you wrote something we would of course consider it. Although,
> perhaps I'm too close to this code but I can't really see a way to
> combine these two features into a single implementation that wouldn't
> magnify the complexity somewhere, either in CouchDB itself, or inside
> the external processes.
>
> On the other hand, the view server code is in desperate need of a make
> over. Adam and I have made attempts independently but neither of us
> has committed anything.
>
>> thanks,
>>
>> Norman
>>
>
> [1] https://github.com/apache/couchdb/blob/trunk/src/couchdb/couch_os_daemons.erl
>

Re: os_daemons and query_servers

[Paul Davis <pa...@gmail.com>]

On Wed, Jan 26, 2011 at 7:03 PM, Norman Barker <no...@gmail.com> wrote:
> Paul,
>
> understand the answer is no (currently), then why?
>

As Randall says, the protocol between the two types of OS processes is
quite different. Each protocol is generally a request/response type of
protocol, except the actors are switched. In the view server code, it
is Erlang making requests to the OS process, and for os_daemons, it is
the OS process that makes the request. Interleaving the two would
undoubtedly cause more headaches for both the Erlang implementation
and anyone wanting to write an OS process that implemented one or both
protocols.

That said, there's not really a whole lot of code in regards to os
daemons. I think you'd be hard pressed to find examples of where we
could share code between the two processes without overgeneralizing
functions to the point where they're harder to reason about. The
entire os_daemons module [1] (which is almost entirely self contained)
weighs in at 364 lines (total, not discounting blank lines or
whitespace).

> a view server can be an external process, python for example, so now
> we have two external managers within couch, why? Why fix a bug in two
> places (I am verifying a timeout bug with the query_servers for an
> external process).

I will give you five internets if you show me a timeout bug that
exists in both the os_daemons code and view_server code.

That said, the os_daemons code was written after the initial view
server implementation using a couple paradigms that I've learned since
last hacking on the view managers. There are definitely bits we could
port back to the view server manager to simplify a lot of logic.

> Is combining os_daemons with external query_servers worth doing, if I
> do this and add a patch can I get this into the trunk?
>

If you wrote something we would of course consider it. Although,
perhaps I'm too close to this code but I can't really see a way to
combine these two features into a single implementation that wouldn't
magnify the complexity somewhere, either in CouchDB itself, or inside
the external processes.

On the other hand, the view server code is in desperate need of a make
over. Adam and I have made attempts independently but neither of us
has committed anything.

> thanks,
>
> Norman
>

[1] https://github.com/apache/couchdb/blob/trunk/src/couchdb/couch_os_daemons.erl

Re: os_daemons and query_servers

[Norman Barker <no...@gmail.com>]

Paul,

understand the answer is no (currently), then why?

a view server can be an external process, python for example, so now
we have two external managers within couch, why? Why fix a bug in two
places (I am verifying a timeout bug with the query_servers for an
external process).

Is combining os_daemons with external query_servers worth doing, if I
do this and add a patch can I get this into the trunk?

thanks,

Norman

On Wed, Jan 26, 2011 at 1:39 PM, Paul Davis <pa...@gmail.com> wrote:
> On Wed, Jan 26, 2011 at 3:11 PM, Norman Barker <no...@gmail.com> wrote:
>> Hi,
>>
>> is there a way to combine query_servers and os_daemons, currently I have
>>
>
> No.
>
>> [query_servers]
>> lang = path/to/interpreter args
>>
>> and I would like
>>
>> [os_daemons]
>> lang_daemon = path/to/interpreter args
>>
>> [query_servers]
>> lang = lang_daemon
>>
>> so as to take advantage of the os_daemons code refactor.
>>
>
> The os daemons aren't related to the view servers. The os daemons are
> a replacement of the db_update_notifiers.
>
>> Is this possible?
>>
>> thanks,
>>
>> Norman
>>
>

Re: os_daemons and query_servers

[Paul Davis <pa...@gmail.com>]

On Wed, Jan 26, 2011 at 3:11 PM, Norman Barker <no...@gmail.com> wrote:
> Hi,
>
> is there a way to combine query_servers and os_daemons, currently I have
>

No.

> [query_servers]
> lang = path/to/interpreter args
>
> and I would like
>
> [os_daemons]
> lang_daemon = path/to/interpreter args
>
> [query_servers]
> lang = lang_daemon
>
> so as to take advantage of the os_daemons code refactor.
>

The os daemons aren't related to the view servers. The os daemons are
a replacement of the db_update_notifiers.

> Is this possible?
>
> thanks,
>
> Norman
>