Re: On Alternative View Engines

[Simon Metson <si...@cloudant.com>]

Hi, 
Would a DSL just define views or also be used to query them (manage start/end key, keys, limit, reduce, group etc.)? What's the goal; something simpler than writing javascript? something more familiar to SQL users? faster view builds? 

What about building something like https://github.com/mongodb/mongo-hadoop?

Worth noting that a DSL didn't come up in Alan's tour of customers. 
Cheers
Simon


On Sunday, 30 June 2013 at 22:47, Robert Newson wrote:

> +1. A natively executed DSL has been on the wishlist for a while now.
> 
> B.
> 
> 
> On 30 June 2013 22:37, Russell Branca <chewbranca@apache.org (mailto:chewbranca@apache.org)> wrote:
> > The discussion of alternative approaches to view engines is one that
> > bubbles up semi regularly, with the latest addition for a Lua native query
> > server described in COUCHDB-1842 by Alexander. Lua is a great language for
> > embedding into systems and provides powerful sandboxing facilities.
> > 
> > I'm very intrigued by optimizing for the standard use case, where a user
> > wants to build a simple secondary index on their data, and then use a built
> > in reduce function. I think we can find a solution that allows a user to
> > define a doc level transformation in a DSL or query language or some other
> > approach that allows us to keep the view generation functionality within
> > the Erlang VM and avoid the overhead costs of using an external engine.
> > 
> > I do think it makes sense to have an external engine for flexibility, and
> > allowing us to focus on the simple cases while providing a fallback for
> > more complex user defined functions.
> > 
> > To experiment with different approaches, I built a Lisp interpreter on top
> > of Erlang with the premise of white listing the entire language, allowing
> > explicit control over what the user can and cannot do in view functions.
> > You can see it here: [Lispenport](https://github.com/chewbranca/lispenport).
> > It's by no means a full solution, but it has some interesting properties
> > such as really just being syntactic sugar on top of Erlang and all
> > constructs are direct Erlang terms, even lambdas are just Erlang funs.
> > 
> > Now, while I would be intrigued by a Lisp DSL for user defined functions in
> > CouchDB, I didn't expect that to be well received by everyone, so I've
> > considered this just an experiment. If we were going to take this approach,
> > I would rather take Lisp Flavored Erlang (LFE, another project by Robert
> > Virding along with Luerl, and also erlog, a Prolog interpreter in Erlang),
> > and rip out all the pieces we would not want a user to access and use LFE
> > as a base starting point. LFE compiles down to intermediate Erlang bytecode
> > and is designed to follow Erlang functionality, making it a nice option for
> > building a view engine to execute in the Erlang VM.
> > 
> > I've also toyed around with the idea of building a NIF around [JQ](
> > http://stedolan.github.io/jq/) which is a great application for slicing and
> > dicing json data structures written in C.
> > 
> > So my general proposal for discussion is that we build a minimal DSL of
> > some sort, providing fast and simple doc manipulations that executes
> > securely in the Erlang VM, and then we abstract out all functionality for a
> > "full" view engine, list functions, show functions, etc to a separate
> > engine that is easily swappable and not required for standard functionality.
> > 
> > Thoughts?
> > 
> > 
> > -Russell 

Re: On Alternative View Engines

[Alexander Shorin <kx...@gmail.com>]

While DSL will be not yet another subset of SQL it's fine (: DSL
should focus on solving specific problem, but not try to adopt some
other solution for it. There are JSON Pointer[1] and JSON Patch[2]
specifications that may be used to play around. I think, this DSL have
to be JSON-driven as CouchDB is.

Anyway, it's very interesting, but also very hard task to create such
DSL that would be simple, effective,
not-allowing-to-shoot-yourself-in-leg and intuitive for the end user.

[1]: http://tools.ietf.org/html/rfc6901
[2]: http://tools.ietf.org/html/rfc6902
--
,,,^..^,,,


On Mon, Jul 1, 2013 at 2:11 PM, Simon Metson <si...@cloudant.com> wrote:
> Hi,
> Would a DSL just define views or also be used to query them (manage start/end key, keys, limit, reduce, group etc.)? What's the goal; something simpler than writing javascript? something more familiar to SQL users? faster view builds?
>
> What about building something like https://github.com/mongodb/mongo-hadoop?
>
> Worth noting that a DSL didn't come up in Alan's tour of customers.
> Cheers
> Simon
>
>
> On Sunday, 30 June 2013 at 22:47, Robert Newson wrote:
>
>> +1. A natively executed DSL has been on the wishlist for a while now.
>>
>> B.
>>
>>
>> On 30 June 2013 22:37, Russell Branca <chewbranca@apache.org (mailto:chewbranca@apache.org)> wrote:
>> > The discussion of alternative approaches to view engines is one that
>> > bubbles up semi regularly, with the latest addition for a Lua native query
>> > server described in COUCHDB-1842 by Alexander. Lua is a great language for
>> > embedding into systems and provides powerful sandboxing facilities.
>> >
>> > I'm very intrigued by optimizing for the standard use case, where a user
>> > wants to build a simple secondary index on their data, and then use a built
>> > in reduce function. I think we can find a solution that allows a user to
>> > define a doc level transformation in a DSL or query language or some other
>> > approach that allows us to keep the view generation functionality within
>> > the Erlang VM and avoid the overhead costs of using an external engine.
>> >
>> > I do think it makes sense to have an external engine for flexibility, and
>> > allowing us to focus on the simple cases while providing a fallback for
>> > more complex user defined functions.
>> >
>> > To experiment with different approaches, I built a Lisp interpreter on top
>> > of Erlang with the premise of white listing the entire language, allowing
>> > explicit control over what the user can and cannot do in view functions.
>> > You can see it here: [Lispenport](https://github.com/chewbranca/lispenport).
>> > It's by no means a full solution, but it has some interesting properties
>> > such as really just being syntactic sugar on top of Erlang and all
>> > constructs are direct Erlang terms, even lambdas are just Erlang funs.
>> >
>> > Now, while I would be intrigued by a Lisp DSL for user defined functions in
>> > CouchDB, I didn't expect that to be well received by everyone, so I've
>> > considered this just an experiment. If we were going to take this approach,
>> > I would rather take Lisp Flavored Erlang (LFE, another project by Robert
>> > Virding along with Luerl, and also erlog, a Prolog interpreter in Erlang),
>> > and rip out all the pieces we would not want a user to access and use LFE
>> > as a base starting point. LFE compiles down to intermediate Erlang bytecode
>> > and is designed to follow Erlang functionality, making it a nice option for
>> > building a view engine to execute in the Erlang VM.
>> >
>> > I've also toyed around with the idea of building a NIF around [JQ](
>> > http://stedolan.github.io/jq/) which is a great application for slicing and
>> > dicing json data structures written in C.
>> >
>> > So my general proposal for discussion is that we build a minimal DSL of
>> > some sort, providing fast and simple doc manipulations that executes
>> > securely in the Erlang VM, and then we abstract out all functionality for a
>> > "full" view engine, list functions, show functions, etc to a separate
>> > engine that is easily swappable and not required for standard functionality.
>> >
>> > Thoughts?
>> >
>> >
>> > -Russell
>
>

Re: On Alternative View Engines

[Noah Slater <ns...@apache.org>]

Thought as much.

Would be very interested in seeing some of that data. Could be good for the
Couch community also!


On 17 August 2013 20:43, Simon Metson <si...@cloudant.com> wrote:

> A failed cross post. We've spoken to Cloudant customers about things
> they'd like to see and a dsl wasn't something they requested.
>
>
> On Saturday, 17 August 2013 at 20:30, Noah Slater wrote:
>
> > Hi Simon,
> >
> > On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
> >
> > > Worth noting that a DSL didn't come up in Alan's tour of customers.
> >
> >
> > You mind me asking what this is a reference to? :)
> >
> > --
> > Noah Slater
> > https://twitter.com/nslater
> >
> >
>
>
>


-- 
Noah Slater
https://twitter.com/nslater

Re: On Alternative View Engines

["nicholas a. evans" <ni...@ekenosen.net>]

For what it's worth, I'm one of those Cloudant customers who spoke to
Alan and told him I wasn't really interested in a DSL.  At the time I
was talking to him, I don't think I realized that a DSL could be a
route to faster view indexing.  DSL to avoid writing javascript: meh,
uninterested.  DSL to enable faster view indexing: very interested,
+1, yes, please.

-- 
Nick

On Sat, Aug 17, 2013 at 3:43 PM, Simon Metson <si...@cloudant.com> wrote:
> A failed cross post. We've spoken to Cloudant customers about things they'd like to see and a dsl wasn't something they requested.
>
>
> On Saturday, 17 August 2013 at 20:30, Noah Slater wrote:
>
>> Hi Simon,
>>
>> On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
>>
>> > Worth noting that a DSL didn't come up in Alan's tour of customers.
>>
>>
>> You mind me asking what this is a reference to? :)
>>
>> --
>> Noah Slater
>> https://twitter.com/nslater
>>
>>
>
>

Re: On Alternative View Engines

[Noah Slater <ns...@apache.org>]

P.S. Thanks! ;)


On 17 August 2013 20:43, Simon Metson <si...@cloudant.com> wrote:

> A failed cross post. We've spoken to Cloudant customers about things
> they'd like to see and a dsl wasn't something they requested.
>
>
> On Saturday, 17 August 2013 at 20:30, Noah Slater wrote:
>
> > Hi Simon,
> >
> > On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
> >
> > > Worth noting that a DSL didn't come up in Alan's tour of customers.
> >
> >
> > You mind me asking what this is a reference to? :)
> >
> > --
> > Noah Slater
> > https://twitter.com/nslater
> >
> >
>
>
>


-- 
Noah Slater
https://twitter.com/nslater

Re: On Alternative View Engines

[Simon Metson <si...@cloudant.com>]

A failed cross post. We've spoken to Cloudant customers about things they'd like to see and a dsl wasn't something they requested.  


On Saturday, 17 August 2013 at 20:30, Noah Slater wrote:

> Hi Simon,
> 
> On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:
> 
> > Worth noting that a DSL didn't come up in Alan's tour of customers.
> 
> 
> You mind me asking what this is a reference to? :)
> 
> -- 
> Noah Slater
> https://twitter.com/nslater
> 
> 

Re: On Alternative View Engines

[Noah Slater <ns...@apache.org>]

Hi Simon,

On 1 July 2013 11:11, Simon Metson <si...@cloudant.com> wrote:

> Worth noting that a DSL didn't come up in Alan's tour of customers.


You mind me asking what this is a reference to? :)

-- 
Noah Slater
https://twitter.com/nslater