You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Johs Ensby <jo...@b2w.com> on 2015/11/10 10:24:39 UTC
Re: [PROPOSAL] Allow rewrites to be JS function
Hi Dale,
the discussions on Couchapps has been an endless circle, so I wanted to write up some thoughts before I answered this.
Also with the new mailing list couchapp@couchdb.apache.org, we will be able to build a discussion on this.
What you propose is clear as crystal if node.js is your platform and you want the fastest way to stich something together. Or you are a master of the npm universe and love big puzzles.
The challenge for CouchDB if node.js is the prerequisite, is that Pouch server becomes the alternative to CouchDB, and MondoDB is a less risky choice for the pure DB. The cluster ambition seems far fetched and with no appeal to front-end developers, the distance to the revenue-generating people increase.
My idea is that with a few tweaks to the very limited 5-year-old application server features of CouchDB, it becomes a platform of integration and could be the center of a new ecosystem. At its present path, CouchDB is on its way to a competitive landscape where the other branch of the *ouch* family is challenging MongoDB, and I think it is a reasonable hedge for Apache CouchDB to nurture the app server features and differentiate as much as possible from Couchbase. If it could be included in the 1.7 release I would be extremely happy. I think it would be a very cheap insurance for the CouchDB project.
Ermouth put together a rewrite function to replace the static rewrite list. I have played with it on a CouchDB 2 build and it makes it extremely easy to do things like
- firewalls
- nested rewrite rules
- regexp based rewrites
- add a modification of your favorute client side router
- user id based rewrites
- peer IP adress based rewrites
- time based rewrite rules
- rewrite rules that make turn get requests to updates
- round robin to multiply CouchDB's request per second capacity
- etc
Of course you can do all of this better elsewhere, but can you install all of it over a single REST request anywhere else than on CouchDB?
I think of the design document to which you point your vhost to as an application server, and it will be very fun for a lot of people that are years away from mastering the tools to get a license to trade.
There is an opportunity here that could render node.js as uncool for a new generation of developers as the LAMP stack is for today's developers.
There is no downside to including ermouth's upgrade to the rewrite, and I believe Alexander is working on that.
Why would we not create a playground for javascript prgrammers inside CouchDB?
It is just an incredible long list of upsides, if anything is a no-brainer it is this one.
To better explain my thinking I am working on a series of articles, the two first here
Applications in CouchDB
- A business strategy
https://medium.com/p/946a9c19015 <https://medium.com/p/946a9c19015>
A possible ecosystem for CouchDB applications
https://medium.com/p/e39ac4397cea <https://medium.com/p/e39ac4397cea>
The last one ends with a plan for a pilot project that identifies 10 different specialized roles in a CouchDB-centric ecosystem where revenue flow back from the paying end user through 5 of the roles. Each role in a hierarchy allows them to specialize on a buseinss relationship with 1-3 other parties.
CouchDB is at the front, PouchDB travels inside as attachment to ddocs, as will any js framwork, node.js is at the back as a service broker.
The main idea: create room for people at the customer-end of the value chain, and lay the foundation for a short multiplication cycle to provide some cash infusion to the CouchDB community to secure its future.
Johs
> On 20. okt. 2015, at 16.05, Dale Harvey <da...@arandomurl.com> wrote:
>
> On the server side you can run what ever JavaScript you want, it is far
> less limited than
> client side JavaScript. PouchDB also runs on the server and has the exact
> same issue
> in terms of seperation of concerns.
>
> PouchDB doesnt expect developers to do dom manipulation, templating,
> scheduling and
> other applications features via API's it provides, neither will it ever
> provide a reverse proxy
>
> Pouch and CouchDB can both better integrate with things that do provide
> reverse proxying
> and other such features much better though. Here is an example of a simple
> server that
> provides a proxy on /proxy, and admin party database on /db and a read only
> database
> on /public (it could use either pouchdb or couchdb to provide the databases)
>
> https://gist.github.com/daleharvey/bf4e9a94553b75ef2d39
>
> Its far from ideal, but already simpler than attempting the same setup with
> CouchDB only
> and far far more extendable. I didnt need to wait for CouchDB to implement
> any features
> or discuss their implementation.
>
> While CouchDB is your only application platform, there will never be enough
> functionality in
> it to fulfill even the basic use cases and people will lean towards better
> solutions
> (see mongo + meteor).
>
> I very much think a single shot application server is a great idea (see
> firebase), however
> I am very much of the opinion that it should be built on top of CouchDB,
> not into it.
>
> On 20 October 2015 at 14:37, Johs. E <jo...@b2w.com> wrote:
>
>> Hi Dale
>> I dont understand the “integrate with their use case as best as we
>> possibly can” when it comes to rewrite and reverse proxy.
>> As far as I know there is nothing to integrate here.
>> The joy for PouchDB to be a pure database is very understandable, since it
>> sits in this rich environment that allow you to do anything.
>> At the server side CouchDB allows for a very limited portion of javaScript
>> processing, it would be good if it was less restrictive.
>> I am not advocating arbitrary application platform functionality.
>> Just take out some limitations for the javascript crowd out there to love
>> CouchDB
>>
>> Thanks and best regards,
>> Johs
>>
>>
>>> On 20 Oct 2015, at 11:10, Dale Harvey <da...@arandomurl.com> wrote:
>>>
>>> This discussion has gone round and round a couple of times in different
>>> forms
>>> so I will avoid repeating my previous points but from working on PouchDB,
>>> the focus on having 'PouchDB' be a database only is fairly liberating, by
>>> not trying
>>> to add fairly arbitrary "application platform" features into the core
>>> codebase we can focus
>>> on integrating with what does provide those application platform features
>>> much better.
>>>
>>> Users do not need to wait for reverse proxying or url rewriting to be an
>>> agreed, implemented
>>> and maintained feature, they can use the many that already exist and we
>>> will ensure
>>> that we integrate with their use case as best as we possibly can.
>>>
>>> On 19 October 2015 at 23:57, Harald Kisch <ha...@gmail.com> wrote:
>>>
>>>> Hi Garren,
>>>>
>>>> look at MSSQL (CLR) or ORACLE (JAVA Forms) any database was trying to
>>>> support their users with markup languages like XML, HTML, etc. for
>> instance
>>>> directly out of the database core (performance, simplicity,
>>>> scalability,..).
>>>> Lotus Notes did also integrate JavaScript inside of their core (Do you
>> know
>>>> which guy did take part of it?). This have different reasons, but one of
>>>> this reasons is to support users with dynamic mutable data directly into
>>>> their GUI in JSON format which in my opinion is the fundamental part of
>>>> CouchDB to be a database for the web.
>>>> Improvements get lost if we look at others and try not to be different.
>> In
>>>> my opinion we should more think about replacing spidermonkey with the
>>>> google V8 engine and itegrate node completely into the CouchDB core to
>>>> consume npm-packages directly instead of using them in the local
>> filesystem
>>>> outside of CouchDB, where unfortunatelly complexity rise up at scaling.
>>>>
>>>> --Harald
>>>>
>>>> On Mon, Oct 19, 2015 at 9:24 PM, Johs Ensby <jo...@b2w.com> wrote:
>>>>
>>>>> Hi Garren,
>>>>> thanks for the "not standing in the way", I hope for more volunteers to
>>>>> iron out some of CouchDB's old akward wrinkles.
>>>>> I am all with you for simplification:) ermouth's rewrite function is a
>>>>> huge simplifier.
>>>>>
>>>>> Where I disagree with you is where you say "probably a sign that this
>>>> idea
>>>>> is not something worth pursuing".
>>>>> Whenever you discover that you have a differentiator, it's always a
>> good
>>>>> idea to look closely before discaring it and blend in with the rest.
>>>>> It's all about attracting the next million web developers.
>>>>>
>>>>> johs
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On 19. okt. 2015, at 20.08, Garren Smith <gs...@redcometlabs.com> wrote:
>>>>>>
>>>>>> I’m really struggling with these proposals. I love the enthusiasm of
>>>>> everyone but I keep thinking we should rather simplify CouchDB.
>>>>>> CouchDB is ultimately a database. One with excellent sync
>> capabilities.
>>>>> And combining that with libraries like PouchDB and Hoodie make it an
>>>>> amazing database to build applications with.
>>>>>> Adding routers and reverse proxies just makes it feel like we trying
>> to
>>>>> push CouchDB into being more than it needs to be.
>>>>>>
>>>>>> For example building Couchapp like functionality in Node.js is so
>>>> simple
>>>>> and way better. Languages like Go also do that really well. Far
>> superior
>>>>> than what we can do with a database.
>>>>>> I would rather let the Node.js and Go web libraries serve content and
>>>>> let us focus on building a clustered replicating database. We will draw
>>>>> more people to this community if we can do that properly over creaky,
>>>> slow
>>>>> and limited web serving mashed on top of a database.
>>>>>> If I look at other popular databases, I don’t see any of them serving
>>>>> web content which is probably a sign that this idea is not something
>>>> worth
>>>>> pursuing.
>>>>>>
>>>>>> However if there is a burning desire for this and developers raising
>>>>> their hands to code this functionality, I would not stand in your way.
>> It
>>>>> is great to see the varied use of CouchDB out in the wild.
>>>>>>
>>>>>> Cheers
>>>>>> Garren
>>>>>>
>>>>>>
>>>>>>> On 19 Oct 2015, at 4:47 PM, Johs. E <jo...@b2w.com> wrote:
>>>>>>>
>>>>>>> Thanks Andy,
>>>>>>> I will try and get some use cases up on confluence.
>>>>>>> As for whoever would pick up the work after ermouth, I have of
>> course
>>>>> one big thing on the wish list that goes well with a new router
>>>> solution..
>>>>>>> reverse proxy
>>>>>>> I remember asking about it when I first started to work w CouchDB and
>>>>> there were some concerns regarding security.
>>>>>>> Since then I think node.js has paved the way with content scraping
>> and
>>>>> all sorts of outgoing traffic.
>>>>>>>
>>>>>>> Has anyone work on a reverse proxy solution for Couch?
>>>>>>>
>>>>>>> johs
>>>>>>>
>>>>>>>
>>>>>>>> On 18 Oct 2015, at 21:36, Andy Wenk <an...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Hey Johs,
>>>>>>>>
>>>>>>>> thanks a lot for this. I need some time to dig into it. We need a
>>>>> place to
>>>>>>>> write the user stories / use case down. So I suggest we find good
>>>>> place at
>>>>>>>> the cwiki. So I suggest to use
>>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/COUCHDB/Enhancement+Proposals
>>>>> .
>>>>>>>> Do you have write access there? If not, please ping me.
>>>>>>>>
>>>>>>>> Great work!
>>>>>>>>
>>>>>>>> All the best
>>>>>>>>
>>>>>>>> Andy
>>>>>>>>
>>>>>>>> P.S.: Jan already mentioned the feature freeze. Please take it not
>>>> as a
>>>>>>>> demotivation but as the possibility to have a bit more time to work
>>>> on
>>>>> it.
>>>>>>>>
>>>>>>>> On 17 October 2015 at 17:32, Johs Ensby <jo...@b2w.com> wrote:
>>>>>>>>
>>>>>>>>> Andy,
>>>>>>>>> I will make my first use case for function in _rewrite a high level
>>>>> one:
>>>>>>>>> to create a standalone server that is an all-in-one DB server,
>>>>> application
>>>>>>>>> server, api server and web server.
>>>>>>>>>
>>>>>>>>> I have played with the build of CouchDB 2 with rewrite function
>>>>>>>>> implemented that ermouth put up on the irish AWS community AMI
>> list
>>>>> and
>>>>>>>>> the new use cases are endless.
>>>>>>>>> First, I find that there are a few things that people fail to
>> notice
>>>>> about
>>>>>>>>> ddocs.
>>>>>>>>> you need a tool to build a ddoc, editing JSON is not a viable
>>>> option.
>>>>> The
>>>>>>>>> Ddoc Lab of ermouth is in a class of its own. If you havent tried
>> it
>>>>> yet,
>>>>>>>>> do so from http://ddoc.me/ <http://ddoc.me/>. Installing on your
>>>> own
>>>>>>>>> couch it is as easy as storing the application, all included as one
>>>>>>>>> document in any database. Ddoc Lab is a component oriented IDE with
>>>>> syntax
>>>>>>>>> checking, less preprosessor and other build tools that let you keep
>>>> a
>>>>> well
>>>>>>>>> organized ddoc as a source project (in one couchdb document) and
>> you
>>>>>>>>> publich a ddoc to any target db.
>>>>>>>>> with this tool you can organize your js modules and templates etc
>>>> and
>>>>>>>>> basically...
>>>>>>>>> set up the API of your application in a ddoc. You can switch
>> between
>>>>>>>>> databases and their ddoc functionality based on username, role or
>>>>>>>>> geolocation and limit access to parts of the Couch API as needed
>>>>>>>>>
>>>>>>>>> This is the method I would recommend to explore powerful simplicity
>>>>> with
>>>>>>>>> function in rewrites
>>>>>>>>> redirect port 80 directly to couch
>>>>>>>>> set up 2 vhosts, one for public access pointing to
>> youdb/_design/api
>>>>> and
>>>>>>>>> one for sysadm pointing to /
>>>>>>>>> for admin use Fauxton and Ddoc Lab on the sysadm vhost
>>>>>>>>> you are set to develop great systems, no big tool stack to learn,
>>>> just
>>>>>>>>> bring in whatever js modules you like, the template engine you
>> like,
>>>>> the
>>>>>>>>> router you like, the HTML5 stuff you like..
>>>>>>>>> .. or just write some very compact js code in one place where you
>>>>> ealier
>>>>>>>>> had to mess around with a whole stack of tools and systems
>>>>>>>>>
>>>>>>>>> below is the req object that the function takes
>>>>>>>>>
>>>>>>>>> Johs
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The rewrite function has this syntax
>>>>>>>>> function(req) {
>>>>>>>>> .. your code that will
>>>>>>>>> return
>>>>>>>>> path
>>>>>>>>> // optional
>>>>>>>>> headers
>>>>>>>>> method // you can change this on the fly
>>>>>>>>> code
>>>>>>>>> body
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> the function receives this req object
>>>>>>>>> method
>>>>>>>>> path
>>>>>>>>> raw_path
>>>>>>>>> query
>>>>>>>>> headers
>>>>>>>>> Accept
>>>>>>>>> Accept-Encoding
>>>>>>>>> Connection
>>>>>>>>> Host
>>>>>>>>> Upgrade-Insecure-Requests
>>>>>>>>> User-Agent
>>>>>>>>> x-couchdb-vhost-path
>>>>>>>>> body
>>>>>>>>> peer
>>>>>>>>> cookie
>>>>>>>>> userCtx
>>>>>>>>> db
>>>>>>>>> name
>>>>>>>>> roles
>>>>>>>>> secObj
>>>>>>>>>
>>>>>>>>>> On 1. okt. 2015, at 13.40, Andy Wenk <an...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Johs,
>>>>>>>>>>
>>>>>>>>>> Yes for sure! That's always great. Maybe you can also write some
>>>> user
>>>>>>>>> stories (given when then) or scribble some graphics. Everything is
>>>>> useful
>>>>>>>>> and will fasten the process ;-)
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>> Andy
>>>>>>>>>>
>>>>>>>>>> On 1 Oct 2015 12:38, "Johs Ensby" <johs@b2w.com <mailto:
>>>> johs@b2w.com
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>> Thanks for this Andy,
>>>>>>>>>>
>>>>>>>>>> I can contribute to the discussion of the feature seen from a user
>>>>>>>>> perspective.
>>>>>>>>>> Would it be appropriate to present some use cases?
>>>>>>>>>>
>>>>>>>>>> best
>>>>>>>>>> Johs
>>>>>>>>>>
>>>>>>>>>>> On 1. okt. 2015, at 12.33, Andy Wenk <andywenk@apache.org
>>>> <mailto:
>>>>>>>>> andywenk@apache.org>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Johs,
>>>>>>>>>>>
>>>>>>>>>>> Let me please show the steps needed.
>>>>>>>>>>>
>>>>>>>>>>> * discuss the feature very clearly on the dev@. Please make sure
>>>>> that
>>>>>>>>> core
>>>>>>>>>>> developers as committers with commit bits are involved
>>>>>>>>>>>
>>>>>>>>>>> * code the feature. Make sure to implement tests
>>>>>>>>>>>
>>>>>>>>>>> * send a pull request and show it to dev@
>>>>>>>>>>>
>>>>>>>>>>> * finally the community will accept or decline the feature (this
>>>>> will
>>>>>>>>>>> involve refactoring and changes)
>>>>>>>>>>>
>>>>>>>>>>> As Alex said. The PMC or Jan do not decide about the feature.
>>>>>>>>>>>
>>>>>>>>>>> All the best
>>>>>>>>>>>
>>>>>>>>>>> Andy
>>>>>>>>>>> On 1 Oct 2015 11:21, "Alexander Shorin" <kxepal@gmail.com
>>>> <mailto:
>>>>>>>>> kxepal@gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Oct 1, 2015 at 12:07 PM, Johs Ensby <johs@b2w.com
>>>> <mailto:
>>>>>>>>> johs@b2w.com>> wrote:
>>>>>>>>>>>>> will you welcome ermouths rewrite contribution?
>>>>>>>>>>>>
>>>>>>>>>>>> The decision is depends on the implementation. If it will be
>>>> good,
>>>>> why
>>>>>>>>>>>> not? Finally, CouchDB is open source project: we cannot forbid
>>>>> people
>>>>>>>>>>>> right for contributions, we only welcome them.
>>>>>>>>>>>>
>>>>>>>>>>>>> Arguments against couchapps has to do with performance and the
>>>>> folly
>>>>>>>>> in
>>>>>>>>>>>> competing with node.js.
>>>>>>>>>>>>
>>>>>>>>>>>> Performance question for the new _rewrite implementation is very
>>>>>>>>>>>> depends on query server. Once it can process this kind of
>>>>> functions,
>>>>>>>>>>>> you may use something better than JS to gain better performance.
>>>>> That
>>>>>>>>>>>> could be Erlang native query server, or luerl-based one, or else
>>>>> you
>>>>>>>>>>>> like.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> ,,,^..^,,,
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Andy Wenk
>>>>>>>> Hamburg - Germany
>>>>>>>> RockIt!
>>>>>>>>
>>>>>>>> GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588
>>>>>>>>
>>>>>>>> https://people.apache.org/keys/committer/andywenk.asc
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>
>>