You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Russell Branca <ch...@apache.org> on 2013/06/30 23:37:32 UTC
On Alternative View Engines
The discussion of alternative approaches to view engines is one that
bubbles up semi regularly, with the latest addition for a Lua native query
server described in COUCHDB-1842 by Alexander. Lua is a great language for
embedding into systems and provides powerful sandboxing facilities.
I'm very intrigued by optimizing for the standard use case, where a user
wants to build a simple secondary index on their data, and then use a built
in reduce function. I think we can find a solution that allows a user to
define a doc level transformation in a DSL or query language or some other
approach that allows us to keep the view generation functionality within
the Erlang VM and avoid the overhead costs of using an external engine.
I do think it makes sense to have an external engine for flexibility, and
allowing us to focus on the simple cases while providing a fallback for
more complex user defined functions.
To experiment with different approaches, I built a Lisp interpreter on top
of Erlang with the premise of white listing the entire language, allowing
explicit control over what the user can and cannot do in view functions.
You can see it here: [Lispenport](https://github.com/chewbranca/lispenport).
It's by no means a full solution, but it has some interesting properties
such as really just being syntactic sugar on top of Erlang and all
constructs are direct Erlang terms, even lambdas are just Erlang funs.
Now, while I would be intrigued by a Lisp DSL for user defined functions in
CouchDB, I didn't expect that to be well received by everyone, so I've
considered this just an experiment. If we were going to take this approach,
I would rather take Lisp Flavored Erlang (LFE, another project by Robert
Virding along with Luerl, and also erlog, a Prolog interpreter in Erlang),
and rip out all the pieces we would not want a user to access and use LFE
as a base starting point. LFE compiles down to intermediate Erlang bytecode
and is designed to follow Erlang functionality, making it a nice option for
building a view engine to execute in the Erlang VM.
I've also toyed around with the idea of building a NIF around [JQ](
http://stedolan.github.io/jq/) which is a great application for slicing and
dicing json data structures written in C.
So my general proposal for discussion is that we build a minimal DSL of
some sort, providing fast and simple doc manipulations that executes
securely in the Erlang VM, and then we abstract out all functionality for a
"full" view engine, list functions, show functions, etc to a separate
engine that is easily swappable and not required for standard functionality.
Thoughts?
-Russell
Re: On Alternative View Engines
Posted by Alexander Shorin <kx...@gmail.com>.
While DSL will be not yet another subset of SQL it's fine (: DSL
should focus on solving specific problem, but not try to adopt some
other solution for it. There are JSON Pointer[1] and JSON Patch[2]
specifications that may be used to play around. I think, this DSL have
to be JSON-driven as CouchDB is.
Anyway, it's very interesting, but also very hard task to create such
DSL that would be simple, effective,
not-allowing-to-shoot-yourself-in-leg and intuitive for the end user.
[1]: http://tools.ietf.org/html/rfc6901
[2]: http://tools.ietf.org/html/rfc6902
--
,,,^..^,,,
On Mon, Jul 1, 2013 at 2:11 PM, Simon Metson <si...@cloudant.com> wrote:
> Hi,
> Would a DSL just define views or also be used to query them (manage start/end key, keys, limit, reduce, group etc.)? What's the goal; something simpler than writing javascript? something more familiar to SQL users? faster view builds?
>
> What about building something like https://github.com/mongodb/mongo-hadoop?
>
> Worth noting that a DSL didn't come up in Alan's tour of customers.
> Cheers
> Simon
>
>
> On Sunday, 30 June 2013 at 22:47, Robert Newson wrote:
>
>> +1. A natively executed DSL has been on the wishlist for a while now.
>>
>> B.
>>
>>
>> On 30 June 2013 22:37, Russell Branca <chewbranca@apache.org (mailto:chewbranca@apache.org)> wrote:
>> > The discussion of alternative approaches to view engines is one that
>> > bubbles up semi regularly, with the latest addition for a Lua native query
>> > server described in COUCHDB-1842 by Alexander. Lua is a great language for
>> > embedding into systems and provides powerful sandboxing facilities.
>> >
>> > I'm very intrigued by optimizing for the standard use case, where a user
>> > wants to build a simple secondary index on their data, and then use a built
>> > in reduce function. I think we can find a solution that allows a user to
>> > define a doc level transformation in a DSL or query language or some other
>> > approach that allows us to keep the view generation functionality within
>> > the Erlang VM and avoid the overhead costs of using an external engine.
>> >
>> > I do think it makes sense to have an external engine for flexibility, and
>> > allowing us to focus on the simple cases while providing a fallback for
>> > more complex user defined functions.
>> >
>> > To experiment with different approaches, I built a Lisp interpreter on top
>> > of Erlang with the premise of white listing the entire language, allowing
>> > explicit control over what the user can and cannot do in view functions.
>> > You can see it here: [Lispenport](https://github.com/chewbranca/lispenport).
>> > It's by no means a full solution, but it has some interesting properties
>> > such as really just being syntactic sugar on top of Erlang and all
>> > constructs are direct Erlang terms, even lambdas are just Erlang funs.
>> >
>> > Now, while I would be intrigued by a Lisp DSL for user defined functions in
>> > CouchDB, I didn't expect that to be well received by everyone, so I've
>> > considered this just an experiment. If we were going to take this approach,
>> > I would rather take Lisp Flavored Erlang (LFE, another project by Robert
>> > Virding along with Luerl, and also erlog, a Prolog interpreter in Erlang),
>> > and rip out all the pieces we would not want a user to access and use LFE
>> > as a base starting point. LFE compiles down to intermediate Erlang bytecode
>> > and is designed to follow Erlang functionality, making it a nice option for
>> > building a view engine to execute in the Erlang VM.
>> >
>> > I've also toyed around with the idea of building a NIF around [JQ](
>> > http://stedolan.github.io/jq/) which is a great application for slicing and
>> > dicing json data structures written in C.
>> >
>> > So my general proposal for discussion is that we build a minimal DSL of
>> > some sort, providing fast and simple doc manipulations that executes
>> > securely in the Erlang VM, and then we abstract out all functionality for a
>> > "full" view engine, list functions, show functions, etc to a separate
>> > engine that is easily swappable and not required for standard functionality.
>> >
>> > Thoughts?
>> >
>> >
>> > -Russell
>
>
Re: On Alternative View Engines
Posted by Noah Slater <ns...@apache.org>.
Thought as much.
Would be very interested in seeing some of that data. Could be good for the
Couch community also!
On 17 August 2013 20:43, Simon Metson <si...@cloudant.com> wrote:
> A failed cross post. We've spoken to Cloudant customers about things
> they'd like to see and a dsl wasn't something they requested.
>
>
> On Saturday, 17 August 2013 at 20:30, Noah Slater wrote:
>
> > Hi Simon,
> >
> > On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
> >
> > > Worth noting that a DSL didn't come up in Alan's tour of customers.
> >
> >
> > You mind me asking what this is a reference to? :)
> >
> > --
> > Noah Slater
> > https://twitter.com/nslater
> >
> >
>
>
>
--
Noah Slater
https://twitter.com/nslater
Re: On Alternative View Engines
Posted by "nicholas a. evans" <ni...@ekenosen.net>.
For what it's worth, I'm one of those Cloudant customers who spoke to
Alan and told him I wasn't really interested in a DSL. At the time I
was talking to him, I don't think I realized that a DSL could be a
route to faster view indexing. DSL to avoid writing javascript: meh,
uninterested. DSL to enable faster view indexing: very interested,
+1, yes, please.
--
Nick
On Sat, Aug 17, 2013 at 3:43 PM, Simon Metson <si...@cloudant.com> wrote:
> A failed cross post. We've spoken to Cloudant customers about things they'd like to see and a dsl wasn't something they requested.
>
>
> On Saturday, 17 August 2013 at 20:30, Noah Slater wrote:
>
>> Hi Simon,
>>
>> On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
>>
>> > Worth noting that a DSL didn't come up in Alan's tour of customers.
>>
>>
>> You mind me asking what this is a reference to? :)
>>
>> --
>> Noah Slater
>> https://twitter.com/nslater
>>
>>
>
>
Re: On Alternative View Engines
Posted by Noah Slater <ns...@apache.org>.
P.S. Thanks! ;)
On 17 August 2013 20:43, Simon Metson <si...@cloudant.com> wrote:
> A failed cross post. We've spoken to Cloudant customers about things
> they'd like to see and a dsl wasn't something they requested.
>
>
> On Saturday, 17 August 2013 at 20:30, Noah Slater wrote:
>
> > Hi Simon,
> >
> > On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
> >
> > > Worth noting that a DSL didn't come up in Alan's tour of customers.
> >
> >
> > You mind me asking what this is a reference to? :)
> >
> > --
> > Noah Slater
> > https://twitter.com/nslater
> >
> >
>
>
>
--
Noah Slater
https://twitter.com/nslater
Re: On Alternative View Engines
Posted by Simon Metson <si...@cloudant.com>.
A failed cross post. We've spoken to Cloudant customers about things they'd like to see and a dsl wasn't something they requested.
On Saturday, 17 August 2013 at 20:30, Noah Slater wrote:
> Hi Simon,
>
> On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
>
> > Worth noting that a DSL didn't come up in Alan's tour of customers.
>
>
> You mind me asking what this is a reference to? :)
>
> --
> Noah Slater
> https://twitter.com/nslater
>
>
Re: On Alternative View Engines
Posted by Noah Slater <ns...@apache.org>.
Hi Simon,
On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
> Worth noting that a DSL didn't come up in Alan's tour of customers.
You mind me asking what this is a reference to? :)
--
Noah Slater
https://twitter.com/nslater
Re: On Alternative View Engines
Posted by Simon Metson <si...@cloudant.com>.
Hi,
Would a DSL just define views or also be used to query them (manage start/end key, keys, limit, reduce, group etc.)? What's the goal; something simpler than writing javascript? something more familiar to SQL users? faster view builds?
What about building something like https://github.com/mongodb/mongo-hadoop?
Worth noting that a DSL didn't come up in Alan's tour of customers.
Cheers
Simon
On Sunday, 30 June 2013 at 22:47, Robert Newson wrote:
> +1. A natively executed DSL has been on the wishlist for a while now.
>
> B.
>
>
> On 30 June 2013 22:37, Russell Branca <chewbranca@apache.org (mailto:chewbranca@apache.org)> wrote:
> > The discussion of alternative approaches to view engines is one that
> > bubbles up semi regularly, with the latest addition for a Lua native query
> > server described in COUCHDB-1842 by Alexander. Lua is a great language for
> > embedding into systems and provides powerful sandboxing facilities.
> >
> > I'm very intrigued by optimizing for the standard use case, where a user
> > wants to build a simple secondary index on their data, and then use a built
> > in reduce function. I think we can find a solution that allows a user to
> > define a doc level transformation in a DSL or query language or some other
> > approach that allows us to keep the view generation functionality within
> > the Erlang VM and avoid the overhead costs of using an external engine.
> >
> > I do think it makes sense to have an external engine for flexibility, and
> > allowing us to focus on the simple cases while providing a fallback for
> > more complex user defined functions.
> >
> > To experiment with different approaches, I built a Lisp interpreter on top
> > of Erlang with the premise of white listing the entire language, allowing
> > explicit control over what the user can and cannot do in view functions.
> > You can see it here: [Lispenport](https://github.com/chewbranca/lispenport).
> > It's by no means a full solution, but it has some interesting properties
> > such as really just being syntactic sugar on top of Erlang and all
> > constructs are direct Erlang terms, even lambdas are just Erlang funs.
> >
> > Now, while I would be intrigued by a Lisp DSL for user defined functions in
> > CouchDB, I didn't expect that to be well received by everyone, so I've
> > considered this just an experiment. If we were going to take this approach,
> > I would rather take Lisp Flavored Erlang (LFE, another project by Robert
> > Virding along with Luerl, and also erlog, a Prolog interpreter in Erlang),
> > and rip out all the pieces we would not want a user to access and use LFE
> > as a base starting point. LFE compiles down to intermediate Erlang bytecode
> > and is designed to follow Erlang functionality, making it a nice option for
> > building a view engine to execute in the Erlang VM.
> >
> > I've also toyed around with the idea of building a NIF around [JQ](
> > http://stedolan.github.io/jq/) which is a great application for slicing and
> > dicing json data structures written in C.
> >
> > So my general proposal for discussion is that we build a minimal DSL of
> > some sort, providing fast and simple doc manipulations that executes
> > securely in the Erlang VM, and then we abstract out all functionality for a
> > "full" view engine, list functions, show functions, etc to a separate
> > engine that is easily swappable and not required for standard functionality.
> >
> > Thoughts?
> >
> >
> > -Russell
Re: On Alternative View Engines
Posted by Robert Newson <rn...@apache.org>.
+1. A natively executed DSL has been on the wishlist for a while now.
B.
On 30 June 2013 22:37, Russell Branca <ch...@apache.org> wrote:
> The discussion of alternative approaches to view engines is one that
> bubbles up semi regularly, with the latest addition for a Lua native query
> server described in COUCHDB-1842 by Alexander. Lua is a great language for
> embedding into systems and provides powerful sandboxing facilities.
>
> I'm very intrigued by optimizing for the standard use case, where a user
> wants to build a simple secondary index on their data, and then use a built
> in reduce function. I think we can find a solution that allows a user to
> define a doc level transformation in a DSL or query language or some other
> approach that allows us to keep the view generation functionality within
> the Erlang VM and avoid the overhead costs of using an external engine.
>
> I do think it makes sense to have an external engine for flexibility, and
> allowing us to focus on the simple cases while providing a fallback for
> more complex user defined functions.
>
> To experiment with different approaches, I built a Lisp interpreter on top
> of Erlang with the premise of white listing the entire language, allowing
> explicit control over what the user can and cannot do in view functions.
> You can see it here: [Lispenport](https://github.com/chewbranca/lispenport).
> It's by no means a full solution, but it has some interesting properties
> such as really just being syntactic sugar on top of Erlang and all
> constructs are direct Erlang terms, even lambdas are just Erlang funs.
>
> Now, while I would be intrigued by a Lisp DSL for user defined functions in
> CouchDB, I didn't expect that to be well received by everyone, so I've
> considered this just an experiment. If we were going to take this approach,
> I would rather take Lisp Flavored Erlang (LFE, another project by Robert
> Virding along with Luerl, and also erlog, a Prolog interpreter in Erlang),
> and rip out all the pieces we would not want a user to access and use LFE
> as a base starting point. LFE compiles down to intermediate Erlang bytecode
> and is designed to follow Erlang functionality, making it a nice option for
> building a view engine to execute in the Erlang VM.
>
> I've also toyed around with the idea of building a NIF around [JQ](
> http://stedolan.github.io/jq/) which is a great application for slicing and
> dicing json data structures written in C.
>
> So my general proposal for discussion is that we build a minimal DSL of
> some sort, providing fast and simple doc manipulations that executes
> securely in the Erlang VM, and then we abstract out all functionality for a
> "full" view engine, list functions, show functions, etc to a separate
> engine that is easily swappable and not required for standard functionality.
>
> Thoughts?
>
>
> -Russell
Re: On Alternative View Engines
Posted by Alexander Shorin <kx...@gmail.com>.
That's right so why it's called JSON pointer - it was design for a bit
different proposes, but still may be useful for DSL/DDL.
--
,,,^..^,,,
On Tue, Jul 2, 2013 at 9:29 PM, Russell Branca <ch...@apache.org> wrote:
> On Tue, Jul 2, 2013 at 10:07 AM, Alexander Shorin <kx...@gmail.com> wrote:
>
>> On Tue, Jul 2, 2013 at 8:31 PM, Russell Branca <ch...@apache.org>
>> wrote:
>> > I like the link to JSON Pointer, Alexander, that's intriguing. Something
>> > like that could be very useful for simple views, although the RFC is
>> quite
>> > vague on edge cases, ie how would you implement something like if
>> (doc.type
>> > == "foo") emit(doc.bar, doc.baz) with JSON pointer. Speaking of JSON
>> > pointer, looks like Jan beat us to the punch ;-)
>> > https://github.com/janl/erl-jsonpointer
>>
>> JSON Pointer suggest to raise an error in case if pointer points to
>> the non existed value.
>> Probably, this case in DDL context will be like:
>>
>> with doc emit /bar, /baz if /type is "foo"
>>
>> loops:
>>
>> with doc map(emit(/bar, /baz) for foo in /boo if /bar is null break)
>>
>> a bit pythonic, but please don't shoot (:
>>
>> --
>> ,,,^..^,,,
>>
>
> Sure, so we can create a DDL or what have you, but unless I'm missing
> something, I don't see a way to make a view that filters on doc.type or
> some other field, and then emits a simple set of values, which I think is
> required for a minimal view implementation. The idea of JSON pointers where
> we can just let users define path strings for the key and value is really
> intriguing, but missing functionality.
>
>
> -Russell
Re: On Alternative View Engines
Posted by Russell Branca <ch...@apache.org>.
On Tue, Jul 2, 2013 at 10:07 AM, Alexander Shorin <kx...@gmail.com> wrote:
> On Tue, Jul 2, 2013 at 8:31 PM, Russell Branca <ch...@apache.org>
> wrote:
> > I like the link to JSON Pointer, Alexander, that's intriguing. Something
> > like that could be very useful for simple views, although the RFC is
> quite
> > vague on edge cases, ie how would you implement something like if
> (doc.type
> > == "foo") emit(doc.bar, doc.baz) with JSON pointer. Speaking of JSON
> > pointer, looks like Jan beat us to the punch ;-)
> > https://github.com/janl/erl-jsonpointer
>
> JSON Pointer suggest to raise an error in case if pointer points to
> the non existed value.
> Probably, this case in DDL context will be like:
>
> with doc emit /bar, /baz if /type is "foo"
>
> loops:
>
> with doc map(emit(/bar, /baz) for foo in /boo if /bar is null break)
>
> a bit pythonic, but please don't shoot (:
>
> --
> ,,,^..^,,,
>
Sure, so we can create a DDL or what have you, but unless I'm missing
something, I don't see a way to make a view that filters on doc.type or
some other field, and then emits a simple set of values, which I think is
required for a minimal view implementation. The idea of JSON pointers where
we can just let users define path strings for the key and value is really
intriguing, but missing functionality.
-Russell
Re: On Alternative View Engines
Posted by Alexander Shorin <kx...@gmail.com>.
On Tue, Jul 2, 2013 at 8:31 PM, Russell Branca <ch...@apache.org> wrote:
> I like the link to JSON Pointer, Alexander, that's intriguing. Something
> like that could be very useful for simple views, although the RFC is quite
> vague on edge cases, ie how would you implement something like if (doc.type
> == "foo") emit(doc.bar, doc.baz) with JSON pointer. Speaking of JSON
> pointer, looks like Jan beat us to the punch ;-)
> https://github.com/janl/erl-jsonpointer
JSON Pointer suggest to raise an error in case if pointer points to
the non existed value.
Probably, this case in DDL context will be like:
with doc emit /bar, /baz if /type is "foo"
loops:
with doc map(emit(/bar, /baz) for foo in /boo if /bar is null break)
a bit pythonic, but please don't shoot (:
--
,,,^..^,,,
Re: On Alternative View Engines
Posted by Russell Branca <ch...@apache.org>.
On Mon, Jul 1, 2013 at 4:36 AM, Filippo Fadda <
filippo.fadda@programmazione.it> wrote:
> I agree with the idea to have a DSL executed inside the Erlang VM. A
> simple, but powerful DDL, not SQL, a strongly typed language just to
> _define_ views, not to query them (that's a different matter). And it
> should have regex support. Actually, to write fast views you need Erlang
> itself and I think having a DDL it's a big plus.
>
> +1
I like the link to JSON Pointer, Alexander, that's intriguing. Something
like that could be very useful for simple views, although the RFC is quite
vague on edge cases, ie how would you implement something like if (doc.type
== "foo") emit(doc.bar, doc.baz) with JSON pointer. Speaking of JSON
pointer, looks like Jan beat us to the punch ;-)
https://github.com/janl/erl-jsonpointer
Filippo, when you say DDL, how feature-ful are you envisioning?
I think the big idea is to get _something_ that functions inside the Erlang
VM securely and efficiently.
-Russell
Re: On Alternative View Engines
Posted by Filippo Fadda <fi...@programmazione.it>.
I agree with the idea to have a DSL executed inside the Erlang VM. A simple, but powerful DDL, not SQL, a strongly typed language just to _define_ views, not to query them (that's a different matter). And it should have regex support. Actually, to write fast views you need Erlang itself and I think having a DDL it's a big plus.
+1
Re: On Alternative View Engines
Posted by Russell Branca <ch...@gmail.com>.
On Tue, Jul 2, 2013 at 2:25 PM, Randall Leeds <ra...@gmail.com>wrote:
> Very cool experiments there.
>
> Have you taken the time to understand JSONiq well enough to say
> whether that would be a good fit?
>
> http://www.jsoniq.org/
>
> On Sun, Jun 30, 2013 at 2:37 PM, Russell Branca <ch...@apache.org>
> wrote:
> > The discussion of alternative approaches to view engines is one that
> > bubbles up semi regularly, with the latest addition for a Lua native
> query
> > server described in COUCHDB-1842 by Alexander. Lua is a great language
> for
> > embedding into systems and provides powerful sandboxing facilities.
> >
> > I'm very intrigued by optimizing for the standard use case, where a user
> > wants to build a simple secondary index on their data, and then use a
> built
> > in reduce function. I think we can find a solution that allows a user to
> > define a doc level transformation in a DSL or query language or some
> other
> > approach that allows us to keep the view generation functionality within
> > the Erlang VM and avoid the overhead costs of using an external engine.
> >
> > I do think it makes sense to have an external engine for flexibility, and
> > allowing us to focus on the simple cases while providing a fallback for
> > more complex user defined functions.
> >
> > To experiment with different approaches, I built a Lisp interpreter on
> top
> > of Erlang with the premise of white listing the entire language, allowing
> > explicit control over what the user can and cannot do in view functions.
> > You can see it here: [Lispenport](
> https://github.com/chewbranca/lispenport).
> > It's by no means a full solution, but it has some interesting properties
> > such as really just being syntactic sugar on top of Erlang and all
> > constructs are direct Erlang terms, even lambdas are just Erlang funs.
> >
> > Now, while I would be intrigued by a Lisp DSL for user defined functions
> in
> > CouchDB, I didn't expect that to be well received by everyone, so I've
> > considered this just an experiment. If we were going to take this
> approach,
> > I would rather take Lisp Flavored Erlang (LFE, another project by Robert
> > Virding along with Luerl, and also erlog, a Prolog interpreter in
> Erlang),
> > and rip out all the pieces we would not want a user to access and use LFE
> > as a base starting point. LFE compiles down to intermediate Erlang
> bytecode
> > and is designed to follow Erlang functionality, making it a nice option
> for
> > building a view engine to execute in the Erlang VM.
> >
> > I've also toyed around with the idea of building a NIF around [JQ](
> > http://stedolan.github.io/jq/) which is a great application for slicing
> and
> > dicing json data structures written in C.
> >
> > So my general proposal for discussion is that we build a minimal DSL of
> > some sort, providing fast and simple doc manipulations that executes
> > securely in the Erlang VM, and then we abstract out all functionality
> for a
> > "full" view engine, list functions, show functions, etc to a separate
> > engine that is easily swappable and not required for standard
> functionality.
> >
> > Thoughts?
> >
> >
> > -Russell
>
Wow JSONiq looks cool! I hadn't seen that before, and it definitely seems
like an intriguing option. Would be a solid chunk of development to make a
new implementation of it for Erlang, but that could work. We would also
want to take a subset of the spec, and not include the pieces of JSONiq
that provide IO, but it definitely looks like it's on the right track for
what we want.
-Russell
Re: On Alternative View Engines
Posted by Randall Leeds <ra...@gmail.com>.
Very cool experiments there.
Have you taken the time to understand JSONiq well enough to say
whether that would be a good fit?
http://www.jsoniq.org/
On Sun, Jun 30, 2013 at 2:37 PM, Russell Branca <ch...@apache.org> wrote:
> The discussion of alternative approaches to view engines is one that
> bubbles up semi regularly, with the latest addition for a Lua native query
> server described in COUCHDB-1842 by Alexander. Lua is a great language for
> embedding into systems and provides powerful sandboxing facilities.
>
> I'm very intrigued by optimizing for the standard use case, where a user
> wants to build a simple secondary index on their data, and then use a built
> in reduce function. I think we can find a solution that allows a user to
> define a doc level transformation in a DSL or query language or some other
> approach that allows us to keep the view generation functionality within
> the Erlang VM and avoid the overhead costs of using an external engine.
>
> I do think it makes sense to have an external engine for flexibility, and
> allowing us to focus on the simple cases while providing a fallback for
> more complex user defined functions.
>
> To experiment with different approaches, I built a Lisp interpreter on top
> of Erlang with the premise of white listing the entire language, allowing
> explicit control over what the user can and cannot do in view functions.
> You can see it here: [Lispenport](https://github.com/chewbranca/lispenport).
> It's by no means a full solution, but it has some interesting properties
> such as really just being syntactic sugar on top of Erlang and all
> constructs are direct Erlang terms, even lambdas are just Erlang funs.
>
> Now, while I would be intrigued by a Lisp DSL for user defined functions in
> CouchDB, I didn't expect that to be well received by everyone, so I've
> considered this just an experiment. If we were going to take this approach,
> I would rather take Lisp Flavored Erlang (LFE, another project by Robert
> Virding along with Luerl, and also erlog, a Prolog interpreter in Erlang),
> and rip out all the pieces we would not want a user to access and use LFE
> as a base starting point. LFE compiles down to intermediate Erlang bytecode
> and is designed to follow Erlang functionality, making it a nice option for
> building a view engine to execute in the Erlang VM.
>
> I've also toyed around with the idea of building a NIF around [JQ](
> http://stedolan.github.io/jq/) which is a great application for slicing and
> dicing json data structures written in C.
>
> So my general proposal for discussion is that we build a minimal DSL of
> some sort, providing fast and simple doc manipulations that executes
> securely in the Erlang VM, and then we abstract out all functionality for a
> "full" view engine, list functions, show functions, etc to a separate
> engine that is easily swappable and not required for standard functionality.
>
> Thoughts?
>
>
> -Russell