You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Stanley Iriele <si...@gmail.com> on 2013/08/14 18:54:46 UTC
Erlang vs JavaScript
I'm curious... What exactly is the performance gain from writing
views/shows/ etc in Erlang vs JavaScript... From what I've seen the
JavaScript view server is sufficiently fast and feature complete.
So my question is this.... Is the Erlang view server that much faster?...
Is there another gain that I'm missing?..and is it as feature complete as
the JavaScript equivalent?
Regards,
Stanley
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
On Sun, Aug 18, 2013 at 1:08 PM, Benoit Chesneau <bc...@gmail.com>wrote:
> On Sunday, August 18, 2013, Jason Smith wrote:
>
> > Nine times faster than O(n) is still O(n).
> >
> > If people take this on, I will be your biggest cheerleader. My point is
> to
> > ask what problem this solves. After some thought I think it does NOT
> solve
> > a technical problem, it solves a perception and adoption problem.
> >
> > The problem a 9x speed up solves is, "I am prototyping something and my
> > benchmarks are not impressive." Maybe that is enough. If CouchDB causes
> > friction for adopters, then this would actually be a big priority.
> >
> > I see a Pareto distribution of data set sizes. Databases are either
> small,
> > with all views building reasonably quickly; or very large (above 10GB)
> > where no speedup will change the fundamental feeling of the view build:
> you
> > kick it off, then you wait. And again, these speeds diminish in
> importance
> > over a project life cycle.
> >
> >
> the problem to solve here is how making the view engine more efficient than
> it is now. Becoming more efficient will naturally make it faster in this
> case.
>
> We can also make it more concurrent or //. o(n) is another problem to
> solve (or not).
>
> on a pure rechnical point faster means faster whatever the feeling is. If
> your 10Mb or 100Mb of data goes over a more efficient or
> faster protocol/process it will be more efficient/faster whatever the
> feeling a human can have. At the end you will also feel better with a
> reduced
> power bill.
>
:)
Yes, I think the community has persuaded me about the merits of a speed
boost.
Re: Erlang vs JavaScript
Posted by Benoit Chesneau <bc...@gmail.com>.
On Sunday, August 18, 2013, Jason Smith wrote:
> Nine times faster than O(n) is still O(n).
>
> If people take this on, I will be your biggest cheerleader. My point is to
> ask what problem this solves. After some thought I think it does NOT solve
> a technical problem, it solves a perception and adoption problem.
>
> The problem a 9x speed up solves is, "I am prototyping something and my
> benchmarks are not impressive." Maybe that is enough. If CouchDB causes
> friction for adopters, then this would actually be a big priority.
>
> I see a Pareto distribution of data set sizes. Databases are either small,
> with all views building reasonably quickly; or very large (above 10GB)
> where no speedup will change the fundamental feeling of the view build: you
> kick it off, then you wait. And again, these speeds diminish in importance
> over a project life cycle.
>
>
the problem to solve here is how making the view engine more efficient than
it is now. Becoming more efficient will naturally make it faster in this
case.
We can also make it more concurrent or //. o(n) is another problem to
solve (or not).
on a pure rechnical point faster means faster whatever the feeling is. If
your 10Mb or 100Mb of data goes over a more efficient or
faster protocol/process it will be more efficient/faster whatever the
feeling a human can have. At the end you will also feel better with a reduced
power bill.
- benoît
> On Sun, Aug 18, 2013 at 11:51 AM, Thanos Vassilakis <thanosv@gmail.com<javascript:;>
> >wrote:
>
> > Build views performance gains:
> > Python 4-6 times faster + less memory
> > Erlang 7-9 times faster + much less memory.
> >
> > On Aug 16, 2013, at 18:54, Jan Lehnardt <jan@apache.org <javascript:;>>
> wrote:
> >
> > > I think it is worth putting Jason’s and Jens’s viewpoints on a scale of
> > “learning to live with the pain” and “finding relief for the pain”, where
> > “pain” is any of View Build Time or View Index Time from Jason’s
> glossary.
> > >
> > > If living with the pain works for you, Jason’s point of view is very
> > valuable because all you need to is accept the current situation :)
> > >
> > > But I think it is also worth considering that there are scenarios where
> > living with the pain is just not a fun or viable thing to do and then
> > finding relief for the pain is a good strategy.
> > >
> > > This thread started a fork on dev@ with concrete engineering
> > suggestions on how to make this a reality in CouchDB, so please join us
> > there if you want to help out.
> > >
> > > Best
> > > Jan
> > > --
> > >
> > >
> > > On Aug 16, 2013, at 17:24 , Jason Smith <jhs@apache.org <javascript:;>>
> wrote:
> > >
> > >> On Fri, Aug 16, 2013 at 10:16 PM, Jens Alfke <jens@couchbase.com<javascript:;>
> >
> > wrote:
> > >>
> > >>>
> > >>> On Aug 15, 2013, at 10:14 AM, Jason Smith <jhs@apache.org<javascript:;>>
> wrote:
> > >>>
> > >>>> To restate my final sentence which you quoted: Accessing a view is
> an
> > >>> index
> > >>>> scan, it hardly matters what the total data size is; therefore after
> > the
> > >>>> building period, all views are basically always instantly available.
> > >>>
> > >>> You’re missing my point, Jason. Introducing an arbitrary “building
> > period”
> > >>> implies you don’t care about the latency between a database update
> and
> > when
> > >>> it becomes visible in view queries. As I said, that may be allowable
> in
> > >>> some types of applications, but definitely not all. To repeat my
> > example,
> > >>> an e-commerce customer is not going to tolerate any “building period”
> > >>> longer than a second or two in between their clicking “Add To Cart”
> > and the
> > >>> item showing up in their cart.
> > >>>
> > >>
> > >> I agree. But a hypothetical "Add to cart" would only be one or two
> > document
> > >> updates. That will be reflected in a view query pretty quickly. The
> HTTP
> > >> overhead is probably still the majority of the time spent; updating
> the
> > >> view index to reflect the new cart will be negligible.
> > >>
> > >> Do we agree there? It occurs to me that you and I may work in
> different
> > >> environments. Perhaps you are accustomed to much heavier update
> > frequency
> > >> than I.
> > >>
> > >> My own glossary:
> > >>
> > >> View build time: Building the views the first time you push a new
> design
> > >> document into an existing database. Can take a very long time, but can
> > be
> > >> done "offline."
> > >>
> > >> View update time: The time it takes couch to update a view to reflect
> > those
> > >> updates which happened since either the view build, or the view update
> > >> (whichever was more recent).
> > >>
> > >> To me, view update time is regrettable but for most applications
> > >> (read-heavy) it is not much of a problem. View build time is not a
> > concern
> > >> because you can run it at your leisure and "promote" it to the true
> > >> document ID using COPY. In most situations, native views solve a
> > >> non-problem (although I've cited a few cromulent exceptions.)
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
Nine times faster than O(n) is still O(n).
If people take this on, I will be your biggest cheerleader. My point is to
ask what problem this solves. After some thought I think it does NOT solve
a technical problem, it solves a perception and adoption problem.
The problem a 9x speed up solves is, "I am prototyping something and my
benchmarks are not impressive." Maybe that is enough. If CouchDB causes
friction for adopters, then this would actually be a big priority.
I see a Pareto distribution of data set sizes. Databases are either small,
with all views building reasonably quickly; or very large (above 10GB)
where no speedup will change the fundamental feeling of the view build: you
kick it off, then you wait. And again, these speeds diminish in importance
over a project life cycle.
On Sun, Aug 18, 2013 at 11:51 AM, Thanos Vassilakis <th...@gmail.com>wrote:
> Build views performance gains:
> Python 4-6 times faster + less memory
> Erlang 7-9 times faster + much less memory.
>
> On Aug 16, 2013, at 18:54, Jan Lehnardt <ja...@apache.org> wrote:
>
> > I think it is worth putting Jason’s and Jens’s viewpoints on a scale of
> “learning to live with the pain” and “finding relief for the pain”, where
> “pain” is any of View Build Time or View Index Time from Jason’s glossary.
> >
> > If living with the pain works for you, Jason’s point of view is very
> valuable because all you need to is accept the current situation :)
> >
> > But I think it is also worth considering that there are scenarios where
> living with the pain is just not a fun or viable thing to do and then
> finding relief for the pain is a good strategy.
> >
> > This thread started a fork on dev@ with concrete engineering
> suggestions on how to make this a reality in CouchDB, so please join us
> there if you want to help out.
> >
> > Best
> > Jan
> > --
> >
> >
> > On Aug 16, 2013, at 17:24 , Jason Smith <jh...@apache.org> wrote:
> >
> >> On Fri, Aug 16, 2013 at 10:16 PM, Jens Alfke <je...@couchbase.com>
> wrote:
> >>
> >>>
> >>> On Aug 15, 2013, at 10:14 AM, Jason Smith <jh...@apache.org> wrote:
> >>>
> >>>> To restate my final sentence which you quoted: Accessing a view is an
> >>> index
> >>>> scan, it hardly matters what the total data size is; therefore after
> the
> >>>> building period, all views are basically always instantly available.
> >>>
> >>> You’re missing my point, Jason. Introducing an arbitrary “building
> period”
> >>> implies you don’t care about the latency between a database update and
> when
> >>> it becomes visible in view queries. As I said, that may be allowable in
> >>> some types of applications, but definitely not all. To repeat my
> example,
> >>> an e-commerce customer is not going to tolerate any “building period”
> >>> longer than a second or two in between their clicking “Add To Cart”
> and the
> >>> item showing up in their cart.
> >>>
> >>
> >> I agree. But a hypothetical "Add to cart" would only be one or two
> document
> >> updates. That will be reflected in a view query pretty quickly. The HTTP
> >> overhead is probably still the majority of the time spent; updating the
> >> view index to reflect the new cart will be negligible.
> >>
> >> Do we agree there? It occurs to me that you and I may work in different
> >> environments. Perhaps you are accustomed to much heavier update
> frequency
> >> than I.
> >>
> >> My own glossary:
> >>
> >> View build time: Building the views the first time you push a new design
> >> document into an existing database. Can take a very long time, but can
> be
> >> done "offline."
> >>
> >> View update time: The time it takes couch to update a view to reflect
> those
> >> updates which happened since either the view build, or the view update
> >> (whichever was more recent).
> >>
> >> To me, view update time is regrettable but for most applications
> >> (read-heavy) it is not much of a problem. View build time is not a
> concern
> >> because you can run it at your leisure and "promote" it to the true
> >> document ID using COPY. In most situations, native views solve a
> >> non-problem (although I've cited a few cromulent exceptions.)
> >
>
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
I agree! At least, assuming you are speaking frankly and not sarcastically.
This is a great moment in the CouchDB community. Lots of viewpoints. Lots
of different experience and expertise. Lots of discussion, all of it civil
and respectful. It was positive for me, I just flat-out changed my mind
midway through, based on argumentation from the community. It feels bracing
and exciting!
On Sun, Aug 18, 2013 at 2:11 PM, Jim Klo <ji...@sri.com> wrote:
> FWIW: this thread has been the best holy war I've read in a long time.
>
> :-)
>
> On Aug 17, 2013, at 11:51 PM, "Alexander Shorin" <kx...@gmail.com> wrote:
>
> > On Sun, Aug 18, 2013 at 10:29 AM, Benoit Chesneau <bc...@gmail.com>
> wrote:
> >> On Sun, Aug 18, 2013 at 6:51 AM, Thanos Vassilakis <thanosv@gmail.com
> >wrote:
> >>
> >>> Build views performance gains:
> >>> Python 4-6 times faster + less memory
> >>
> >>
> >> Whatever the results of this benchmark are, this not the first time, I
> >> heard that the python engine is faster than the Javascript one. I wonder
> >> what could make it faster than the javascript engine. Since the protocol
> >> and encode/decode steps are the same, this is likely an issue in the
> >> couchjs program. That would be interesting to see where the JS engine
> lost
> >> times and eventually fix it.
> >
> > As for my experience, Python is not faster than JS until you don't
> > have to use stdlib or other packages actively. Lack of stdlib power
> > forces you write/ship a lot of routine code, that mostly not so fast
> > and effective as some C-extension on disk "that does magic" for
> > Python. For simple cases like
> >
> > def map(doc):
> > if 'foo' in doc:
> > yield doc['foo'], 1
> >
> > you will not see the profit.
> >
> > --
> > ,,,^..^,,,
>
Re: Erlang vs JavaScript
Posted by Jim Klo <ji...@sri.com>.
FWIW: this thread has been the best holy war I've read in a long time.
:-)
On Aug 17, 2013, at 11:51 PM, "Alexander Shorin" <kx...@gmail.com> wrote:
> On Sun, Aug 18, 2013 at 10:29 AM, Benoit Chesneau <bc...@gmail.com> wrote:
>> On Sun, Aug 18, 2013 at 6:51 AM, Thanos Vassilakis <th...@gmail.com>wrote:
>>
>>> Build views performance gains:
>>> Python 4-6 times faster + less memory
>>
>>
>> Whatever the results of this benchmark are, this not the first time, I
>> heard that the python engine is faster than the Javascript one. I wonder
>> what could make it faster than the javascript engine. Since the protocol
>> and encode/decode steps are the same, this is likely an issue in the
>> couchjs program. That would be interesting to see where the JS engine lost
>> times and eventually fix it.
>
> As for my experience, Python is not faster than JS until you don't
> have to use stdlib or other packages actively. Lack of stdlib power
> forces you write/ship a lot of routine code, that mostly not so fast
> and effective as some C-extension on disk "that does magic" for
> Python. For simple cases like
>
> def map(doc):
> if 'foo' in doc:
> yield doc['foo'], 1
>
> you will not see the profit.
>
> --
> ,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
On Sun, Aug 18, 2013 at 10:29 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Sun, Aug 18, 2013 at 6:51 AM, Thanos Vassilakis <th...@gmail.com>wrote:
>
>> Build views performance gains:
>> Python 4-6 times faster + less memory
>
>
> Whatever the results of this benchmark are, this not the first time, I
> heard that the python engine is faster than the Javascript one. I wonder
> what could make it faster than the javascript engine. Since the protocol
> and encode/decode steps are the same, this is likely an issue in the
> couchjs program. That would be interesting to see where the JS engine lost
> times and eventually fix it.
As for my experience, Python is not faster than JS until you don't
have to use stdlib or other packages actively. Lack of stdlib power
forces you write/ship a lot of routine code, that mostly not so fast
and effective as some C-extension on disk "that does magic" for
Python. For simple cases like
def map(doc):
if 'foo' in doc:
yield doc['foo'], 1
you will not see the profit.
--
,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Benoit Chesneau <bc...@gmail.com>.
On Sun, Aug 18, 2013 at 6:51 AM, Thanos Vassilakis <th...@gmail.com>wrote:
> Build views performance gains:
> Python 4-6 times faster + less memory
Whatever the results of this benchmark are, this not the first time, I
heard that the python engine is faster than the Javascript one. I wonder
what could make it faster than the javascript engine. Since the protocol
and encode/decode steps are the same, this is likely an issue in the
couchjs program. That would be interesting to see where the JS engine lost
times and eventually fix it.
> On Aug 16, 2013, at 18:54, Jan Lehnardt <ja...@apache.org> wrote:
>
> > I think it is worth putting Jason’s and Jens’s viewpoints on a scale of
> “learning to live with the pain” and “finding relief for the pain”, where
> “pain” is any of View Build Time or View Index Time from Jason’s glossary.
> >
> > If living with the pain works for you, Jason’s point of view is very
> valuable because all you need to is accept the current situation :)
> >
> > But I think it is also worth considering that there are scenarios where
> living with the pain is just not a fun or viable thing to do and then
> finding relief for the pain is a good strategy.
> >
> > This thread started a fork on dev@ with concrete engineering
> suggestions on how to make this a reality in CouchDB, so please join us
> there if you want to help out.
> >
> > Best
> > Jan
> > --
> >
> >
> > On Aug 16, 2013, at 17:24 , Jason Smith <jh...@apache.org> wrote:
> >
> >> On Fri, Aug 16, 2013 at 10:16 PM, Jens Alfke <je...@couchbase.com>
> wrote:
> >>
> >>>
> >>> On Aug 15, 2013, at 10:14 AM, Jason Smith <jh...@apache.org> wrote:
> >>>
> >>>> To restate my final sentence which you quoted: Accessing a view is an
> >>> index
> >>>> scan, it hardly matters what the total data size is; therefore after
> the
> >>>> building period, all views are basically always instantly available.
> >>>
> >>> You’re missing my point, Jason. Introducing an arbitrary “building
> period”
> >>> implies you don’t care about the latency between a database update and
> when
> >>> it becomes visible in view queries. As I said, that may be allowable in
> >>> some types of applications, but definitely not all. To repeat my
> example,
> >>> an e-commerce customer is not going to tolerate any “building period”
> >>> longer than a second or two in between their clicking “Add To Cart”
> and the
> >>> item showing up in their cart.
> >>>
> >>
> >> I agree. But a hypothetical "Add to cart" would only be one or two
> document
> >> updates. That will be reflected in a view query pretty quickly. The HTTP
> >> overhead is probably still the majority of the time spent; updating the
> >> view index to reflect the new cart will be negligible.
> >>
> >> Do we agree there? It occurs to me that you and I may work in different
> >> environments. Perhaps you are accustomed to much heavier update
> frequency
> >> than I.
> >>
> >> My own glossary:
> >>
> >> View build time: Building the views the first time you push a new design
> >> document into an existing database. Can take a very long time, but can
> be
> >> done "offline."
> >>
> >> View update time: The time it takes couch to update a view to reflect
> those
> >> updates which happened since either the view build, or the view update
> >> (whichever was more recent).
> >>
> >> To me, view update time is regrettable but for most applications
> >> (read-heavy) it is not much of a problem. View build time is not a
> concern
> >> because you can run it at your leisure and "promote" it to the true
> >> document ID using COPY. In most situations, native views solve a
> >> non-problem (although I've cited a few cromulent exceptions.)
> >
>
Re: Erlang vs JavaScript
Posted by Thanos Vassilakis <th...@gmail.com>.
Build views performance gains:
Python 4-6 times faster + less memory
Erlang 7-9 times faster + much less memory.
On Aug 16, 2013, at 18:54, Jan Lehnardt <ja...@apache.org> wrote:
> I think it is worth putting Jason’s and Jens’s viewpoints on a scale of “learning to live with the pain” and “finding relief for the pain”, where “pain” is any of View Build Time or View Index Time from Jason’s glossary.
>
> If living with the pain works for you, Jason’s point of view is very valuable because all you need to is accept the current situation :)
>
> But I think it is also worth considering that there are scenarios where living with the pain is just not a fun or viable thing to do and then finding relief for the pain is a good strategy.
>
> This thread started a fork on dev@ with concrete engineering suggestions on how to make this a reality in CouchDB, so please join us there if you want to help out.
>
> Best
> Jan
> --
>
>
> On Aug 16, 2013, at 17:24 , Jason Smith <jh...@apache.org> wrote:
>
>> On Fri, Aug 16, 2013 at 10:16 PM, Jens Alfke <je...@couchbase.com> wrote:
>>
>>>
>>> On Aug 15, 2013, at 10:14 AM, Jason Smith <jh...@apache.org> wrote:
>>>
>>>> To restate my final sentence which you quoted: Accessing a view is an
>>> index
>>>> scan, it hardly matters what the total data size is; therefore after the
>>>> building period, all views are basically always instantly available.
>>>
>>> You’re missing my point, Jason. Introducing an arbitrary “building period”
>>> implies you don’t care about the latency between a database update and when
>>> it becomes visible in view queries. As I said, that may be allowable in
>>> some types of applications, but definitely not all. To repeat my example,
>>> an e-commerce customer is not going to tolerate any “building period”
>>> longer than a second or two in between their clicking “Add To Cart” and the
>>> item showing up in their cart.
>>>
>>
>> I agree. But a hypothetical "Add to cart" would only be one or two document
>> updates. That will be reflected in a view query pretty quickly. The HTTP
>> overhead is probably still the majority of the time spent; updating the
>> view index to reflect the new cart will be negligible.
>>
>> Do we agree there? It occurs to me that you and I may work in different
>> environments. Perhaps you are accustomed to much heavier update frequency
>> than I.
>>
>> My own glossary:
>>
>> View build time: Building the views the first time you push a new design
>> document into an existing database. Can take a very long time, but can be
>> done "offline."
>>
>> View update time: The time it takes couch to update a view to reflect those
>> updates which happened since either the view build, or the view update
>> (whichever was more recent).
>>
>> To me, view update time is regrettable but for most applications
>> (read-heavy) it is not much of a problem. View build time is not a concern
>> because you can run it at your leisure and "promote" it to the true
>> document ID using COPY. In most situations, native views solve a
>> non-problem (although I've cited a few cromulent exceptions.)
>
Re: Erlang vs JavaScript
Posted by Jan Lehnardt <ja...@apache.org>.
I think it is worth putting Jason’s and Jens’s viewpoints on a scale of “learning to live with the pain” and “finding relief for the pain”, where “pain” is any of View Build Time or View Index Time from Jason’s glossary.
If living with the pain works for you, Jason’s point of view is very valuable because all you need to is accept the current situation :)
But I think it is also worth considering that there are scenarios where living with the pain is just not a fun or viable thing to do and then finding relief for the pain is a good strategy.
This thread started a fork on dev@ with concrete engineering suggestions on how to make this a reality in CouchDB, so please join us there if you want to help out.
Best
Jan
--
On Aug 16, 2013, at 17:24 , Jason Smith <jh...@apache.org> wrote:
> On Fri, Aug 16, 2013 at 10:16 PM, Jens Alfke <je...@couchbase.com> wrote:
>
>>
>> On Aug 15, 2013, at 10:14 AM, Jason Smith <jh...@apache.org> wrote:
>>
>>> To restate my final sentence which you quoted: Accessing a view is an
>> index
>>> scan, it hardly matters what the total data size is; therefore after the
>>> building period, all views are basically always instantly available.
>>
>> You’re missing my point, Jason. Introducing an arbitrary “building period”
>> implies you don’t care about the latency between a database update and when
>> it becomes visible in view queries. As I said, that may be allowable in
>> some types of applications, but definitely not all. To repeat my example,
>> an e-commerce customer is not going to tolerate any “building period”
>> longer than a second or two in between their clicking “Add To Cart” and the
>> item showing up in their cart.
>>
>
> I agree. But a hypothetical "Add to cart" would only be one or two document
> updates. That will be reflected in a view query pretty quickly. The HTTP
> overhead is probably still the majority of the time spent; updating the
> view index to reflect the new cart will be negligible.
>
> Do we agree there? It occurs to me that you and I may work in different
> environments. Perhaps you are accustomed to much heavier update frequency
> than I.
>
> My own glossary:
>
> View build time: Building the views the first time you push a new design
> document into an existing database. Can take a very long time, but can be
> done "offline."
>
> View update time: The time it takes couch to update a view to reflect those
> updates which happened since either the view build, or the view update
> (whichever was more recent).
>
> To me, view update time is regrettable but for most applications
> (read-heavy) it is not much of a problem. View build time is not a concern
> because you can run it at your leisure and "promote" it to the true
> document ID using COPY. In most situations, native views solve a
> non-problem (although I've cited a few cromulent exceptions.)
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
On Fri, Aug 16, 2013 at 10:16 PM, Jens Alfke <je...@couchbase.com> wrote:
>
> On Aug 15, 2013, at 10:14 AM, Jason Smith <jh...@apache.org> wrote:
>
> > To restate my final sentence which you quoted: Accessing a view is an
> index
> > scan, it hardly matters what the total data size is; therefore after the
> > building period, all views are basically always instantly available.
>
> You’re missing my point, Jason. Introducing an arbitrary “building period”
> implies you don’t care about the latency between a database update and when
> it becomes visible in view queries. As I said, that may be allowable in
> some types of applications, but definitely not all. To repeat my example,
> an e-commerce customer is not going to tolerate any “building period”
> longer than a second or two in between their clicking “Add To Cart” and the
> item showing up in their cart.
>
I agree. But a hypothetical "Add to cart" would only be one or two document
updates. That will be reflected in a view query pretty quickly. The HTTP
overhead is probably still the majority of the time spent; updating the
view index to reflect the new cart will be negligible.
Do we agree there? It occurs to me that you and I may work in different
environments. Perhaps you are accustomed to much heavier update frequency
than I.
My own glossary:
View build time: Building the views the first time you push a new design
document into an existing database. Can take a very long time, but can be
done "offline."
View update time: The time it takes couch to update a view to reflect those
updates which happened since either the view build, or the view update
(whichever was more recent).
To me, view update time is regrettable but for most applications
(read-heavy) it is not much of a problem. View build time is not a concern
because you can run it at your leisure and "promote" it to the true
document ID using COPY. In most situations, native views solve a
non-problem (although I've cited a few cromulent exceptions.)
Re: Erlang vs JavaScript
Posted by Jens Alfke <je...@couchbase.com>.
On Aug 15, 2013, at 10:14 AM, Jason Smith <jh...@apache.org> wrote:
> To restate my final sentence which you quoted: Accessing a view is an index
> scan, it hardly matters what the total data size is; therefore after the
> building period, all views are basically always instantly available.
You’re missing my point, Jason. Introducing an arbitrary “building period” implies you don’t care about the latency between a database update and when it becomes visible in view queries. As I said, that may be allowable in some types of applications, but definitely not all. To repeat my example, an e-commerce customer is not going to tolerate any “building period” longer than a second or two in between their clicking “Add To Cart” and the item showing up in their cart.
—Jens
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
Sorry if this is beating a dead horse.
On Thu, Aug 15, 2013 at 11:37 AM, Jens Alfke <je...@couchbase.com> wrote:
>
> On Aug 14, 2013, at 8:52 PM, Jason Smith <jh...@apache.org> wrote:
>
> > View building time feels important during development but it is
> irrelevant
> > during operation. Production servers never hold up requests to build
> views,
> > so the performance is secondary.
>
> Wait, what? I may be mis-reading that, but it doesn’t make sense to me, at
> least not in the absolute terms you’re using. I can think of plenty of
> situations where an app server needs an up-to-date view query to get the
> correct data, and can’t just get away with using stale=true.
>
To restate my final sentence which you quoted: Accessing a view is an index
scan, it hardly matters what the total data size is; therefore after the
building period, all views are basically always instantly available.
And my point was, if you use the PPP trick I described (and most production
applications should be, IOMO), then every view, even hundreds of gigabytes,
comes on-line in an instant and querying it is log(n) or whatever. So when
I said "never" I should say "average delay approaches zero as time ->
infinity" or in other words "building views sucks, but you should do that
offline, so who cares?"
The only place where my point fails is that CouchDB updates views not when
you provide new data, but when you query for it. Most applications are
read-heavy, so this is not so bad, but sometimes I wish CouchDB would not
send me a 201 until the views were current. That is one of the few places
CouchDB differs from Google App Engine.
As a practical matter, if somebody said their views do not update fast
enough, I would give them my usual chestnut: your hardware is underpowered.
How many writes can possibly arrive that hang up your view queries so badly?
Unfortunately bitter experience has taught me the answer. About ten
thousand. If you get ten thousand updates (which is easy with bulk_docs,
but even an untuned server can nearly handle about that many concurrent
connections), then you get appreciable downtime on your views. Sad but true.
And in my experience ?stale=ok is just turning a simple problem into a
complex problem multiplied by a scheduling problem.
So that is the only exception I would make but my basic point is, build
views offline and COPY them into production and for all your applications
can tell, they are up instantly. Hope that clarifies my point.
Re: Erlang vs JavaScript
Posted by Andrey Kuprianov <an...@gmail.com>.
I see what you mean now. Thing is that we are actively moving from MySQL to
Couch now and migrating tons of data at a time. So in return all the views
need to catch up with all of those new documents. Erlang seems to be a
promising candidate to speed up our development. That's all. I just tried
it yesterday and I'm pretty happy with how much faster it is than
Javascript.
On Thu, Aug 15, 2013 at 12:01 PM, Jens Alfke <je...@couchbase.com> wrote:
>
> On Aug 14, 2013, at 9:53 PM, Andrey Kuprianov <an...@gmail.com>
> wrote:
>
> > Nope, not misreading anything. And I understand they do. But the sooner
> > they finish the better, wont you agree? It's mainly because of that we
> dont
> > do view updates during weekdays.
>
> OK, you must be using CouchDB for some kind of analytics, or other batch
> processing, if you can get away for _days_ with stale views. Which is a
> totally valid and important use of CouchDB. But it’s completely different
> from using it in a live user-facing application, where changes need to be
> reflected pretty near instantly. That’s why I’m disagreeing with Jason’s
> unqualified statements.
>
> If anyone needs an example, consider a typical online shopping site. Let’s
> say the page that shows the contents of your shopping cart is driven by a
> view query. If a customer adds an item to their cart and then goes to view
> their cart a few seconds later, the item had better show up. If it takes 30
> seconds to show up — let alone a week — that’s going to confuse and annoy
> users.
>
> —Jens
Re: Erlang vs JavaScript
Posted by Jens Alfke <je...@couchbase.com>.
On Aug 14, 2013, at 9:53 PM, Andrey Kuprianov <an...@gmail.com> wrote:
> Nope, not misreading anything. And I understand they do. But the sooner
> they finish the better, wont you agree? It's mainly because of that we dont
> do view updates during weekdays.
OK, you must be using CouchDB for some kind of analytics, or other batch processing, if you can get away for _days_ with stale views. Which is a totally valid and important use of CouchDB. But it’s completely different from using it in a live user-facing application, where changes need to be reflected pretty near instantly. That’s why I’m disagreeing with Jason’s unqualified statements.
If anyone needs an example, consider a typical online shopping site. Let’s say the page that shows the contents of your shopping cart is driven by a view query. If a customer adds an item to their cart and then goes to view their cart a few seconds later, the item had better show up. If it takes 30 seconds to show up — let alone a week — that’s going to confuse and annoy users.
—Jens
Re: Erlang vs JavaScript
Posted by Andrey Kuprianov <an...@gmail.com>.
Nope, not misreading anything. And I understand they do. But the sooner
they finish the better, wont you agree? It's mainly because of that we dont
do view updates during weekdays. So we normally wait until Sunday, in order
to make sure that no users are affected by degraded server performance.
@Jason was also right to mention development, which becomes a waste of time
when you start working with views.
On Thu, Aug 15, 2013 at 11:37 AM, Jens Alfke <je...@couchbase.com> wrote:
>
> On Aug 14, 2013, at 8:52 PM, Jason Smith <jh...@apache.org> wrote:
>
> > View building time feels important during development but it is
> irrelevant
> > during operation. Production servers never hold up requests to build
> views,
> > so the performance is secondary.
>
> Wait, what? I may be mis-reading that, but it doesn’t make sense to me, at
> least not in the absolute terms you’re using. I can think of plenty of
> situations where an app server needs an up-to-date view query to get the
> correct data, and can’t just get away with using stale=true.
>
> —Jens
>
>
Re: Erlang vs JavaScript
Posted by Jens Alfke <je...@couchbase.com>.
On Aug 14, 2013, at 8:52 PM, Jason Smith <jh...@apache.org> wrote:
> View building time feels important during development but it is irrelevant
> during operation. Production servers never hold up requests to build views,
> so the performance is secondary.
Wait, what? I may be mis-reading that, but it doesn’t make sense to me, at least not in the absolute terms you’re using. I can think of plenty of situations where an app server needs an up-to-date view query to get the correct data, and can’t just get away with using stale=true.
—Jens
Re: Erlang vs JavaScript
Posted by Robert Keizer <ro...@keizer.ca>.
On 13-08-18 09:33 AM, Alexander Shorin wrote:
> On Sun, Aug 18, 2013 at 3:54 PM, Volker Mische <vo...@gmail.com> wrote:
>> On 08/18/2013 08:42 AM, Alexander Shorin wrote:
>>> On Sun, Aug 18, 2013 at 10:22 AM, Benoit Chesneau <bc...@gmail.com> wrote:
>>>> On Fri, Aug 16, 2013 at 9:58 PM, Alexander Shorin <kx...@gmail.com> wrote:
>>>>
>>>>> On Fri, Aug 16, 2013 at 11:23 PM, Jason Smith <jh...@apache.org> wrote:
>>>>>> On Fri, Aug 16, 2013 at 4:49 PM, Volker Mische <vo...@gmail.com>
>>>>>> wrote:
>>>>>>> On 08/16/2013 11:32 AM, Alexander Shorin wrote:
>>>>>>>> On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bchesneau@gmail.com
>>>>>>>> wrote:
>>>>>>>>> I agree, (modulo the fact that I would replace a string by a binary
>>>>> ;)
>>>>>>>>> but
>>>>>>>>> that would be only possible if we extract the metadata (_id, _rev)
>>>>> from
>>>>>>>>> the
>>>>>>>>> JSON so couchdb wouldn't have to decode the JSON to get them.
>>>>> Streaming
>>>>>>>>> json would also allows that but since there is no guaranty in the
>>>>>>>>> properties order of a JSON it would be less efficient.
>>>>>>>> What if we split document metadata from document itself?
>>>>>>
>>>>>> I would like to hear a goal for this effort? What is the definition of
>>>>>> success and failure?
>>>>> Idea: move document metadata into separate object.
>>>>>
>>>> How do you link the metadata to the separate object there? Do you let the
>>>> application set the internal links?
>>>>
>>>> I'm +1 with such idea anyway.
>>> Mmm...how I imagine it (Disclaimer: I'm sure I'm wrong in details there!):
>>>
>>> Btree:
>>>
>>> ----+----
>>> | |
>>> --+-- --+--
>>> | | | |
>>> * * * *
>>>
>>> At the node we have doc object {...} for specific revision. Instead of
>>> this, we'll have a tuple ({...}, {...}) - first is a meta, second is a
>>> data.
>>> So I think there wouldn't be needed internal links since meta and data
>>> would live within same Btree node.
>>> For regular doc requesting, they will be merged (still need for `_`
>>> prefix to avoid collisions?) and returned as single {...} as always.
>> We could also return them as separate objects, so the view function
>> becomes: function(doc, meta) {}.
>>
>> Couchbase does that and from my experience it works well and feel right.
> Oh, so this idea even works (:
>
> However, the trick was about to not pass doc part (in case if it big
> enough) to the view server until view server wouldn't process his
> metadata. Otherwise this is good feature, but it wouldn't help with
> indexing speed up. I remind the trick: first process meta part and if
> it passed - load the doc. Later I'd sent another mail where I'd
> eventually reinvented chained views, because trick with meta does
> exactly the same, chained views are more correct way to go. See quote
> at the end with resume.
>
> Anyway, I feel we need to inherit Couchbase experience with document's
> metadata object (of course if they wouldn't sue us for that ((: )
> since everyone already same some preferred metadata fields (like type)
> or uses special object for that to not pollute main document body.
> I'm prefer special '.meta' object at the document root which holds
> document type info, authorship, timestamps, bindings, etc.
> It's good feature to have no matter does it optimizes indexation
> process or not (:
I would suggest either prefixing with an underscore, or the use of a
separate object passed to the view server.
If someone ( such as myself ) has many many documents, which happen to
contain a "meta" attribute, it would be non-trivial to upgrade /
migrate. A migration script could be written of course, although it
wouldn't be ideal;
Something to consider, it may be worth while to simply use obj._meta
instead of .meta.
>
> Below is about chained views:
>
> On Fri, Aug 16, 2013 at 11:58 PM, Alexander Shorin <kx...@gmail.com> wrote:
>> Resume: probably, I'd just described chained views feature with
>> autoindexing by certain fields (:
>> Removing autoindexing feature and we could make views building process
>> much more faster if we make right views chain which will use set
>> algebra operations to calculate target doc ids to pass to final view:
>> reduce docs before map results:
>>
>> {
>> "views": {
>> "posts": {"map": "...", "reduce": "..."},
>> "chain": [
>> ["by_type", {"key": "post"}],
>> ["hidden", {"key": false}],
>> ["by_domain", {"keys": ["public", "wiki"]}]
>> ]
>> }
>> }
>>
>> In case of 10000 docs db with 1200 posts where 200 are hidden and 400
>> are private, result view posts have to process only 600 docs instead
>> of 10000 and it's index lookup operation to find out the result docs
>> to pass. Sure, calling such view triggers all views in the chain.
>
Chained views would be awesome! I'm sure I'm not alone in having solved
this problem by using multiple queries and matching document IDs.
Re: Erlang vs JavaScript
Posted by JFC Morfin <je...@jefsey.com>.
At 11:49 16/08/2013, Volker Mische wrote:
> > What if we split document metadata from document itself? E.g. pass
> > _id, _rev and other system or meta fields with separate object. Their
> > size much lesser than whole document, so it will be possible to fast
> > decode this metadata and decide is doc need to be processed or not
> > without need to decode/encode megabytes of document's json. Sure, this
> > adds additional communication roundtrip, but in case if it will be
> > faster than json decode/encode - why not?
>
>That would be the ultimate-ultimate goal.
This is a basic requirement for me: incrementally (i.e. metadata on
metadata) and for syllodata (data between data) interlinks.
jfc
Re: Erlang vs JavaScript
Posted by "nicholas a. evans" <ni...@ekenosen.net>.
It seems like there might be several simple "internalizing" speedups,
even before tackling the view server protocol or the couchjs view
server, hinted at by Alexander's suggestion:
On Fri, Aug 16, 2013 at 3:58 PM, Alexander Shorin <kx...@gmail.com> wrote:
> Idea: move document metadata into separate object.
...
> Case 2: Large docs. Profit in case when you have set right fields into
> metadata (like doc type, authorship, tags etc.) and filter first by
> this metadata - you have minimal memory footprint, you have less CPU
> load, rule "fast accept - fast reject" works perfectly.
For the simple case of filtering which fields are passed to the map
fn, you don't need full blown chained views, you only need a simple
way to define field filters (describing which fields are the relevant
"metadata" fields).
> Side effect: it's possible to autoindex metadata on fly on document
> update without asking user to write (meta/by_type, meta/by_author,
> meta/by_update_time etc. viiews) . Sure, as much metadata you have as
> large base index will be. In 80% cases it will be no more than 4KB.
Similarly to how the internals of couch already optimize away the case
when you have multiple views in the same design doc share the same map
function (but different reductions), we should also be able to
optimize away the case where multiple views share the same fields
filter.
> Resume: probably, I'd just described chained views feature with
> autoindexing by certain fields (:
One lesson I learned when I looked into implementing chained
map/reduce views is that they will need to be in different design_docs
from the parent views, in order to play nicely with BigCouch. Keeping
them in the same design_doc just doesn't work with parallel view
builds (at least, not without breaking normal design_doc
considerations). So although I really like the simplicity of the
"keep chained views in one design doc" approach, it's probably a
dead-end.
> Removing autoindexing feature and we could make views building process
> much more faster if we make right views chain which will use set
> algebra operations to calculate target doc ids to pass to final view:
> reduce docs before map results:
>
> {
> "views": {
> "posts": {"map": "...", "reduce": "..."},
> "chain": [
> ["by_type", {"key": "post"}],
> ["hidden", {"key": false}],
> ["by_domain", {"keys": ["public", "wiki"]}]
> ]
> }
> }
I was inspired by your view syntax and thought I'd put forward my own
similar proposal:
{
"_id": "plain_old_views_for_comparison",
"views": {
"single_emit": {
"map": "function(doc) { if (!doc.foo) { emit([doc.bar, doc.baz],
doc.quux); } }",
"reduce": "_count"
},
"multiple_emits": {
"map": "function(doc) { if (!doc.foo) { emit([0, doc.bar],
doc.quux); emit(['baz', doc.baz], doc.quux); } }",
"reduce": "_count"
},
}
{
"_id": "internalized",
"options": {
"filter": "!foo",
"fields": ["bar", "baz", "quux"]
},
"views": {
"single_emit_1": {
"map": "function(doc) { emit([doc.bar, doc.baz], doc.quux); }",
"reduce": "_count"
},
"single_emit_2": {
"map": { "key": ["bar", "baz"], "value": "quux" },
"reduce": "_count"
},
"multiple_emits": {
"map": { "emits": [[0, "bar"], "quux"], ["'baz'", "baz"], "quux"]] },
"reduce": "_count"
},
}
Where the above views should behave the same way. The view options
would support "filter" as a guard clause and "fields" to strip out all
but the relevant metadata. These should be defined at the design
document level to simplify working with the current view server
protocol. And the view "map" could optionally be an object describing
the emit values instead of a function string.
The filter string should be simple but powerful: I'd suggest
supporting !, &&, ||, (), "foo.bar.baz", >, <, >=, <=, ==, !=,
numbers, and strings (for "type == 'foo'"). But even if all it
supported was "foo" and "!foo", it would still be useful. In some
cases, this will prevent most docs from ever needing to be evaluated
by the view server. The "fields" array might also consider filtering
nested fields like with "foo.bar.baz". The "filter" and internal map
("key", "value", "emits") should support the same values that "fields"
supports plus numbers and strings, or they could support the same
syntax as "filter" to do things like "key": ["!!deleted_at",
"deleted_at"]. The "filter" and internal map would be able to use all
of the fields, not just the ones defined in the options.
Another odd case where I've personally noticed indexing speed get
immensely bogged down is when the reduce function merges the map
objects together. I've seen views with this problem go up to 5GB
during initial load and compact back down to 20MB. I've documented
this problem and my workaround here:
https://gist.github.com/nevans/5512593. The hideously reduce pattern
in that gist has resulted in 2-5x faster view builds for me (small DBs
infinitesimally slower, huge speedup for big DBs). But it would be
*much* better to simply add a "minimum_reduced_group_level" option to
the view, and let the Erlang handle that without doing unnecessary
view server round trips and hideously complicated reduce functions.
Any group_level below the minimum_reduced_group_level would simply
return "null" for all of the values.
This isn't a trivial proposal, but it can be implemented completely
independently of any view server protocol or couchjs changes. And
even a simplified version could still yield major speedup for some of
the most common map patterns, just as "_sum" and "_count" speed up the
most common reduce functions. Also, the individual pieces can be
implemented independently: If I were to work on this myself (probably
not going to happen in the next month or two), I'd do
"minimum_reduced_group_level" first and "filter" second, since I think
that's were *my* biggest bang for the buck would be. But other
people's dataset (e.g. large docs) might get the biggest improvement
from "fields". And if you have lots of simple map functions, you
might get the biggest speedup from the internal map "key", "value",
"emits".
What do you think? Ugly and untenable? Or a shot in the right direction?
Also, I know that Jason already yielded on the O(N) argument, but I
got here late and wanted to add my $0.02: Obviously anything better
than O(N) is impossible when you need to map N documents. Changing to
O(N/Q) (where Q=parallelism of view indexing; e.g. throw hardware at
it) is still essentially O(N), but it's very useful and something that
BigCouch does nicely. A 9x speedup might be the difference between a
rollout taking 90 hours (barely finishes over the weekend) and 10
hours (you can do it overnight during the week). The longer the view
rollout period, the slower and more cautious the
development/deployment cycle becomes. More importantly, it might be
the difference between loading a large user in 9 hours vs 60 minutes,
which will feel like a qualitative improvement to that user and is
especially important when that user is e.g. Walt Mossberg and load
time is one of two nitpicks he has in his review. Or when you have a
hundred similarly jumbo-sized users sign up the next day. Sorry for
piling on after the argument is over. :)
--
Nick
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
On Sun, Aug 18, 2013 at 3:54 PM, Volker Mische <vo...@gmail.com> wrote:
> On 08/18/2013 08:42 AM, Alexander Shorin wrote:
>> On Sun, Aug 18, 2013 at 10:22 AM, Benoit Chesneau <bc...@gmail.com> wrote:
>>> On Fri, Aug 16, 2013 at 9:58 PM, Alexander Shorin <kx...@gmail.com> wrote:
>>>
>>>> On Fri, Aug 16, 2013 at 11:23 PM, Jason Smith <jh...@apache.org> wrote:
>>>>> On Fri, Aug 16, 2013 at 4:49 PM, Volker Mische <vo...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> On 08/16/2013 11:32 AM, Alexander Shorin wrote:
>>>>>>> On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bchesneau@gmail.com
>>>>>
>>>>>>> wrote:
>>>>>>>> I agree, (modulo the fact that I would replace a string by a binary
>>>> ;)
>>>>>>>> but
>>>>>>>> that would be only possible if we extract the metadata (_id, _rev)
>>>> from
>>>>>>>> the
>>>>>>>> JSON so couchdb wouldn't have to decode the JSON to get them.
>>>> Streaming
>>>>>>>> json would also allows that but since there is no guaranty in the
>>>>>>>> properties order of a JSON it would be less efficient.
>>>>>>>
>>>>>>> What if we split document metadata from document itself?
>>>>>
>>>>>
>>>>> I would like to hear a goal for this effort? What is the definition of
>>>>> success and failure?
>>>>
>>>> Idea: move document metadata into separate object.
>>>>
>>>
>>> How do you link the metadata to the separate object there? Do you let the
>>> application set the internal links?
>>>
>>> I'm +1 with such idea anyway.
>>
>> Mmm...how I imagine it (Disclaimer: I'm sure I'm wrong in details there!):
>>
>> Btree:
>>
>> ----+----
>> | |
>> --+-- --+--
>> | | | |
>> * * * *
>>
>> At the node we have doc object {...} for specific revision. Instead of
>> this, we'll have a tuple ({...}, {...}) - first is a meta, second is a
>> data.
>> So I think there wouldn't be needed internal links since meta and data
>> would live within same Btree node.
>> For regular doc requesting, they will be merged (still need for `_`
>> prefix to avoid collisions?) and returned as single {...} as always.
>
> We could also return them as separate objects, so the view function
> becomes: function(doc, meta) {}.
>
> Couchbase does that and from my experience it works well and feel right.
Oh, so this idea even works (:
However, the trick was about to not pass doc part (in case if it big
enough) to the view server until view server wouldn't process his
metadata. Otherwise this is good feature, but it wouldn't help with
indexing speed up. I remind the trick: first process meta part and if
it passed - load the doc. Later I'd sent another mail where I'd
eventually reinvented chained views, because trick with meta does
exactly the same, chained views are more correct way to go. See quote
at the end with resume.
Anyway, I feel we need to inherit Couchbase experience with document's
metadata object (of course if they wouldn't sue us for that ((: )
since everyone already same some preferred metadata fields (like type)
or uses special object for that to not pollute main document body.
I'm prefer special '.meta' object at the document root which holds
document type info, authorship, timestamps, bindings, etc.
It's good feature to have no matter does it optimizes indexation
process or not (:
Below is about chained views:
On Fri, Aug 16, 2013 at 11:58 PM, Alexander Shorin <kx...@gmail.com> wrote:
> Resume: probably, I'd just described chained views feature with
> autoindexing by certain fields (:
> Removing autoindexing feature and we could make views building process
> much more faster if we make right views chain which will use set
> algebra operations to calculate target doc ids to pass to final view:
> reduce docs before map results:
>
> {
> "views": {
> "posts": {"map": "...", "reduce": "..."},
> "chain": [
> ["by_type", {"key": "post"}],
> ["hidden", {"key": false}],
> ["by_domain", {"keys": ["public", "wiki"]}]
> ]
> }
> }
>
> In case of 10000 docs db with 1200 posts where 200 are hidden and 400
> are private, result view posts have to process only 600 docs instead
> of 10000 and it's index lookup operation to find out the result docs
> to pass. Sure, calling such view triggers all views in the chain.
--
,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Volker Mische <vo...@gmail.com>.
On 08/18/2013 08:42 AM, Alexander Shorin wrote:
> On Sun, Aug 18, 2013 at 10:22 AM, Benoit Chesneau <bc...@gmail.com> wrote:
>> On Fri, Aug 16, 2013 at 9:58 PM, Alexander Shorin <kx...@gmail.com> wrote:
>>
>>> On Fri, Aug 16, 2013 at 11:23 PM, Jason Smith <jh...@apache.org> wrote:
>>>> On Fri, Aug 16, 2013 at 4:49 PM, Volker Mische <vo...@gmail.com>
>>>> wrote:
>>>>>
>>>>> On 08/16/2013 11:32 AM, Alexander Shorin wrote:
>>>>>> On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bchesneau@gmail.com
>>>>
>>>>>> wrote:
>>>>>>> I agree, (modulo the fact that I would replace a string by a binary
>>> ;)
>>>>>>> but
>>>>>>> that would be only possible if we extract the metadata (_id, _rev)
>>> from
>>>>>>> the
>>>>>>> JSON so couchdb wouldn't have to decode the JSON to get them.
>>> Streaming
>>>>>>> json would also allows that but since there is no guaranty in the
>>>>>>> properties order of a JSON it would be less efficient.
>>>>>>
>>>>>> What if we split document metadata from document itself?
>>>>
>>>>
>>>> I would like to hear a goal for this effort? What is the definition of
>>>> success and failure?
>>>
>>> Idea: move document metadata into separate object.
>>>
>>
>> How do you link the metadata to the separate object there? Do you let the
>> application set the internal links?
>>
>> I'm +1 with such idea anyway.
>
> Mmm...how I imagine it (Disclaimer: I'm sure I'm wrong in details there!):
>
> Btree:
>
> ----+----
> | |
> --+-- --+--
> | | | |
> * * * *
>
> At the node we have doc object {...} for specific revision. Instead of
> this, we'll have a tuple ({...}, {...}) - first is a meta, second is a
> data.
> So I think there wouldn't be needed internal links since meta and data
> would live within same Btree node.
> For regular doc requesting, they will be merged (still need for `_`
> prefix to avoid collisions?) and returned as single {...} as always.
We could also return them as separate objects, so the view function
becomes: function(doc, meta) {}.
Couchbase does that and from my experience it works well and feel right.
Cheers,
Volker
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
On Sun, Aug 18, 2013 at 10:22 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Fri, Aug 16, 2013 at 9:58 PM, Alexander Shorin <kx...@gmail.com> wrote:
>
>> On Fri, Aug 16, 2013 at 11:23 PM, Jason Smith <jh...@apache.org> wrote:
>> > On Fri, Aug 16, 2013 at 4:49 PM, Volker Mische <vo...@gmail.com>
>> > wrote:
>> >>
>> >> On 08/16/2013 11:32 AM, Alexander Shorin wrote:
>> >> > On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bchesneau@gmail.com
>> >
>> >> > wrote:
>> >> >> I agree, (modulo the fact that I would replace a string by a binary
>> ;)
>> >> >> but
>> >> >> that would be only possible if we extract the metadata (_id, _rev)
>> from
>> >> >> the
>> >> >> JSON so couchdb wouldn't have to decode the JSON to get them.
>> Streaming
>> >> >> json would also allows that but since there is no guaranty in the
>> >> >> properties order of a JSON it would be less efficient.
>> >> >
>> >> > What if we split document metadata from document itself?
>> >
>> >
>> > I would like to hear a goal for this effort? What is the definition of
>> > success and failure?
>>
>> Idea: move document metadata into separate object.
>>
>
> How do you link the metadata to the separate object there? Do you let the
> application set the internal links?
>
> I'm +1 with such idea anyway.
Mmm...how I imagine it (Disclaimer: I'm sure I'm wrong in details there!):
Btree:
----+----
| |
--+-- --+--
| | | |
* * * *
At the node we have doc object {...} for specific revision. Instead of
this, we'll have a tuple ({...}, {...}) - first is a meta, second is a
data.
So I think there wouldn't be needed internal links since meta and data
would live within same Btree node.
For regular doc requesting, they will be merged (still need for `_`
prefix to avoid collisions?) and returned as single {...} as always.
--
,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, Aug 16, 2013 at 9:58 PM, Alexander Shorin <kx...@gmail.com> wrote:
> On Fri, Aug 16, 2013 at 11:23 PM, Jason Smith <jh...@apache.org> wrote:
> > On Fri, Aug 16, 2013 at 4:49 PM, Volker Mische <vo...@gmail.com>
> > wrote:
> >>
> >> On 08/16/2013 11:32 AM, Alexander Shorin wrote:
> >> > On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bchesneau@gmail.com
> >
> >> > wrote:
> >> >> I agree, (modulo the fact that I would replace a string by a binary
> ;)
> >> >> but
> >> >> that would be only possible if we extract the metadata (_id, _rev)
> from
> >> >> the
> >> >> JSON so couchdb wouldn't have to decode the JSON to get them.
> Streaming
> >> >> json would also allows that but since there is no guaranty in the
> >> >> properties order of a JSON it would be less efficient.
> >> >
> >> > What if we split document metadata from document itself?
> >
> >
> > I would like to hear a goal for this effort? What is the definition of
> > success and failure?
>
> Idea: move document metadata into separate object.
>
How do you link the metadata to the separate object there? Do you let the
application set the internal links?
I'm +1 with such idea anyway.
> Motivation:
>
> Case 1: Small docs. No profit at all. More over, probably it's better
> to not split things there e.g. pass full doc if his size around some
> amount of megabytes.
> Case 2: Large docs. Profit in case when you have set right fields into
> metadata (like doc type, authorship, tags etc.) and filter first by
> this metadata - you have minimal memory footprint, you have less CPU
> load, rule "fast accept - fast reject" works perfectly.
>
> Side effect: it's possible to first filter by metadata and leave only
> required to process document ids. And if we known what and how many to
> process, we may make assumptions about parallel indexation.
>
> Side effect: it's possible to autoindex metadata on fly on document
> update without asking user to write (meta/by_type, meta/by_author,
> meta/by_update_time etc. viiews) . Sure, as much metadata you have as
> large base index will be. In 80% cases it will be no more than 4KB.
>
> Resume: probably, I'd just described chained views feature with
> autoindexing by certain fields (:
> Removing autoindexing feature and we could make views building process
> much more faster if we make right views chain which will use set
> algebra operations to calculate target doc ids to pass to final view:
> reduce docs before map results:
>
> {
> "views": {
> "posts": {"map": "...", "reduce": "..."},
> "chain": [
> ["by_type", {"key": "post"}],
> ["hidden", {"key": false}],
> ["by_domain", {"keys": ["public", "wiki"]}]
> ]
> }
> }
>
> In case of 10000 docs db with 1200 posts where 200 are hidden and 400
> are private, result view posts have to process only 600 docs instead
> of 10000 and it's index lookup operation to find out the result docs
> to pass. Sure, calling such view triggers all views in the chain. And
> I don't think about cross dependencies and loops for know.
>
> --
> ,,,^..^,,,
>
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
On Fri, Aug 16, 2013 at 11:23 PM, Jason Smith <jh...@apache.org> wrote:
> On Fri, Aug 16, 2013 at 4:49 PM, Volker Mische <vo...@gmail.com>
> wrote:
>>
>> On 08/16/2013 11:32 AM, Alexander Shorin wrote:
>> > On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bc...@gmail.com>
>> > wrote:
>> >> I agree, (modulo the fact that I would replace a string by a binary ;)
>> >> but
>> >> that would be only possible if we extract the metadata (_id, _rev) from
>> >> the
>> >> JSON so couchdb wouldn't have to decode the JSON to get them. Streaming
>> >> json would also allows that but since there is no guaranty in the
>> >> properties order of a JSON it would be less efficient.
>> >
>> > What if we split document metadata from document itself?
>
>
> I would like to hear a goal for this effort? What is the definition of
> success and failure?
Idea: move document metadata into separate object.
Motivation:
Case 1: Small docs. No profit at all. More over, probably it's better
to not split things there e.g. pass full doc if his size around some
amount of megabytes.
Case 2: Large docs. Profit in case when you have set right fields into
metadata (like doc type, authorship, tags etc.) and filter first by
this metadata - you have minimal memory footprint, you have less CPU
load, rule "fast accept - fast reject" works perfectly.
Side effect: it's possible to first filter by metadata and leave only
required to process document ids. And if we known what and how many to
process, we may make assumptions about parallel indexation.
Side effect: it's possible to autoindex metadata on fly on document
update without asking user to write (meta/by_type, meta/by_author,
meta/by_update_time etc. viiews) . Sure, as much metadata you have as
large base index will be. In 80% cases it will be no more than 4KB.
Resume: probably, I'd just described chained views feature with
autoindexing by certain fields (:
Removing autoindexing feature and we could make views building process
much more faster if we make right views chain which will use set
algebra operations to calculate target doc ids to pass to final view:
reduce docs before map results:
{
"views": {
"posts": {"map": "...", "reduce": "..."},
"chain": [
["by_type", {"key": "post"}],
["hidden", {"key": false}],
["by_domain", {"keys": ["public", "wiki"]}]
]
}
}
In case of 10000 docs db with 1200 posts where 200 are hidden and 400
are private, result view posts have to process only 600 docs instead
of 10000 and it's index lookup operation to find out the result docs
to pass. Sure, calling such view triggers all views in the chain. And
I don't think about cross dependencies and loops for know.
--
,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
On Fri, Aug 16, 2013 at 4:49 PM, Volker Mische <vo...@gmail.com>wrote:
> On 08/16/2013 11:32 AM, Alexander Shorin wrote:
> > On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bc...@gmail.com>
> wrote:
> >> I agree, (modulo the fact that I would replace a string by a binary ;)
> but
> >> that would be only possible if we extract the metadata (_id, _rev) from
> the
> >> JSON so couchdb wouldn't have to decode the JSON to get them. Streaming
> >> json would also allows that but since there is no guaranty in the
> >> properties order of a JSON it would be less efficient.
> >
> > What if we split document metadata from document itself?
>
I would like to hear a goal for this effort? What is the definition of
success and failure?
Jan makes a fine point on user@. I "live with the pain." But really, life
is pain. Deny it if you must. Until we are delivered--finally!--our sweet
release, we will necessarily endure pain.
Facts:
* When you store a record, a machine must write that to storage
* If you have an index, a machine must update the index to storage
Building an index requires visiting every document. One way or another, the
entire .couch file is coming off the disk and going through the ringer. One
way or another, every row in the view will be written. I am not clear why
optimizing from N ms/doc to 1/2 N ms/doc will help, when you still have to
read 30GB from storage, and write 30GB back.
One one end the computer scientist says we cannot avoid the necessary time
complexity. On the other end, the casual user says, if it is not
instantaneous, then it hardly matters.
That is, we have a problem of expectation management, not codec speed.
Nobody expects MySQL's CREATE INDEX to finish in a flash, and nobody should
expect that of a view.
If somebody does set out to accelerate views, you're welcome. But I would
ask: what is a successful optimization, and why?
(Also, Noah, if you are out there, this is an example of the sort of thing
I would put on the wiki but past bad experiences make me say "can't be
bothered.")
Re: Erlang vs JavaScript
Posted by JFC Morfin <je...@jefsey.com>.
At 11:49 16/08/2013, Volker Mische wrote:
> > What if we split document metadata from document itself? E.g. pass
> > _id, _rev and other system or meta fields with separate object. Their
> > size much lesser than whole document, so it will be possible to fast
> > decode this metadata and decide is doc need to be processed or not
> > without need to decode/encode megabytes of document's json. Sure, this
> > adds additional communication roundtrip, but in case if it will be
> > faster than json decode/encode - why not?
>
>That would be the ultimate-ultimate goal.
This is a basic requirement for me: incrementally (i.e. metadata on
metadata) and for syllodata (data between data) interlinks.
jfc
Re: Erlang vs JavaScript
Posted by Volker Mische <vo...@gmail.com>.
On 08/16/2013 11:32 AM, Alexander Shorin wrote:
> On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>> I agree, (modulo the fact that I would replace a string by a binary ;) but
>> that would be only possible if we extract the metadata (_id, _rev) from the
>> JSON so couchdb wouldn't have to decode the JSON to get them. Streaming
>> json would also allows that but since there is no guaranty in the
>> properties order of a JSON it would be less efficient.
>
> What if we split document metadata from document itself? E.g. pass
> _id, _rev and other system or meta fields with separate object. Their
> size much lesser than whole document, so it will be possible to fast
> decode this metadata and decide is doc need to be processed or not
> without need to decode/encode megabytes of document's json. Sure, this
> adds additional communication roundtrip, but in case if it will be
> faster than json decode/encode - why not?
That would be the ultimate-ultimate goal.
Cheers,
Volker
Re: Erlang vs JavaScript
Posted by Volker Mische <vo...@gmail.com>.
On 08/16/2013 11:32 AM, Alexander Shorin wrote:
> On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>> I agree, (modulo the fact that I would replace a string by a binary ;) but
>> that would be only possible if we extract the metadata (_id, _rev) from the
>> JSON so couchdb wouldn't have to decode the JSON to get them. Streaming
>> json would also allows that but since there is no guaranty in the
>> properties order of a JSON it would be less efficient.
>
> What if we split document metadata from document itself? E.g. pass
> _id, _rev and other system or meta fields with separate object. Their
> size much lesser than whole document, so it will be possible to fast
> decode this metadata and decide is doc need to be processed or not
> without need to decode/encode megabytes of document's json. Sure, this
> adds additional communication roundtrip, but in case if it will be
> faster than json decode/encode - why not?
That would be the ultimate-ultimate goal.
Cheers,
Volker
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> I agree, (modulo the fact that I would replace a string by a binary ;) but
> that would be only possible if we extract the metadata (_id, _rev) from the
> JSON so couchdb wouldn't have to decode the JSON to get them. Streaming
> json would also allows that but since there is no guaranty in the
> properties order of a JSON it would be less efficient.
What if we split document metadata from document itself? E.g. pass
_id, _rev and other system or meta fields with separate object. Their
size much lesser than whole document, so it will be possible to fast
decode this metadata and decide is doc need to be processed or not
without need to decode/encode megabytes of document's json. Sure, this
adds additional communication roundtrip, but in case if it will be
faster than json decode/encode - why not?
--
,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
On Fri, Aug 16, 2013 at 1:12 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> I agree, (modulo the fact that I would replace a string by a binary ;) but
> that would be only possible if we extract the metadata (_id, _rev) from the
> JSON so couchdb wouldn't have to decode the JSON to get them. Streaming
> json would also allows that but since there is no guaranty in the
> properties order of a JSON it would be less efficient.
What if we split document metadata from document itself? E.g. pass
_id, _rev and other system or meta fields with separate object. Their
size much lesser than whole document, so it will be possible to fast
decode this metadata and decide is doc need to be processed or not
without need to decode/encode megabytes of document's json. Sure, this
adds additional communication roundtrip, but in case if it will be
faster than json decode/encode - why not?
--
,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, Aug 16, 2013 at 11:05 AM, Volker Mische <vo...@gmail.com>wrote:
> On 08/15/2013 11:53 AM, Benoit Chesneau wrote:
> > On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <ja...@apache.org> wrote:
> >
> >>
> >> On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
> >>
> >>> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
> >>> could have shipped an erlang view server that worked in a separate
> >>> process and had the stdio overhead, to combine the slowness of the
> >>> protocol with the obtuseness of erlang. ;)
> >>>
> >>> Evaluating Javascript within the erlang VM process intrigues me, Jens,
> >>> how is that done in your case? I've not previously found the assertion
> >>> that V8 would be faster than SpiderMonkey for a view server compelling
> >>> since the bottleneck is almost never in the code evaluation, but I do
> >>> support CouchDB switching to it for the synergy effects of a closer
> >>> binding with node.js, but if it's running in the same process, that
> >>> would change (though I don't immediately see why the same couldn't be
> >>> done for SpiderMonkey). Off the top of my head, I don't know a safe
> >>> way to evaluate JS in the VM. A NIF-based approach would either be
> >>> quite elaborate or would trip all the scheduling problems that
> >>> long-running NIF's are now notorious for.
> >>>
> >>> At a step removed, the view server protocol itself seems like the
> >>> thing to improve on, it feels like that's the principal bottleneck.
> >>
> >> The code is here:
> >> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
> >>
> >> I’d love for someone to pick this up and give CouchDB, say, a
> ./configure
> >> --enable-native-v8 option or a plugin that allows people to opt into the
> >> speed improvements made there. :)
> >>
> >> The choice for V8 was made because of easier integration API and more
> >> reliable releases as a standalone project, which I think was a smart
> move.
> >>
> >> IIRC it relies on a change to CouchDB-y internals that has not made it
> >> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s
> reading
> >> this thread), but we should look into that and get us “native JS
> views”, at
> >> least as an option or plugin.
> >>
> >> CCing dev@.
> >>
> >> Jan
> >> --
> >>
> >>
> > Well on the first hand nifs look like a good idea but can be very
> > problematic:
> >
> > - when the view computation take time it would block the full vm
> > scheduling. It can be mitigated using a pool of threads to execute the
> work
> > asynchronously but then can create other problems like memory leaking
> etc.
> > - nifs can't be upgraded easily during hot upgrade
> > - when a nif crash, all the vm crash.
> >
> > (Note that we have the same problem when using a nif to decode/encode
> json,
> > it only works well with medium sized documents)
> >
> > One other way to improve the js handling would be removing the main
> > bottleneck ie the serialization-deserialization we do on each step. Not
> > sure if it exists but feasible, why not passing erlang terms from erlang
> > to js and js to erlang? So at the end the deserialization would happen
> only
> > on the JS side ie instead of having
> >
> > get erlang term
> > encode to json
> > send to js
> > decode json
> > process
> > encode json
> > send json
> > decode json to erlang term
> > store
> >
> > we sould just have
> >
> > get erlang term
> > send over STDIO
> > decode erlang term to JS object
> > process
> > encode to erlang term
> > send erlang term
> > store
> >
> > Erlang serialization is also very optimised.
>
> I think the ultimate goal should be to be as little
> conversion/serialisation as possible, hence no conversion to Erlang
> Terms at all.
>
> Input as string
> Parsing to get ID
> Store as string
>
> Send to JS as string
> Process with JS
> Store as string
>
> Cheers,
> Volker
>
>
>
I agree, (modulo the fact that I would replace a string by a binary ;) but
that would be only possible if we extract the metadata (_id, _rev) from the
JSON so couchdb wouldn't have to decode the JSON to get them. Streaming
json would also allows that but since there is no guaranty in the
properties order of a JSON it would be less efficient.
- benoit
Re: Erlang vs JavaScript
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, Aug 16, 2013 at 11:05 AM, Volker Mische <vo...@gmail.com>wrote:
> On 08/15/2013 11:53 AM, Benoit Chesneau wrote:
> > On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <ja...@apache.org> wrote:
> >
> >>
> >> On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
> >>
> >>> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
> >>> could have shipped an erlang view server that worked in a separate
> >>> process and had the stdio overhead, to combine the slowness of the
> >>> protocol with the obtuseness of erlang. ;)
> >>>
> >>> Evaluating Javascript within the erlang VM process intrigues me, Jens,
> >>> how is that done in your case? I've not previously found the assertion
> >>> that V8 would be faster than SpiderMonkey for a view server compelling
> >>> since the bottleneck is almost never in the code evaluation, but I do
> >>> support CouchDB switching to it for the synergy effects of a closer
> >>> binding with node.js, but if it's running in the same process, that
> >>> would change (though I don't immediately see why the same couldn't be
> >>> done for SpiderMonkey). Off the top of my head, I don't know a safe
> >>> way to evaluate JS in the VM. A NIF-based approach would either be
> >>> quite elaborate or would trip all the scheduling problems that
> >>> long-running NIF's are now notorious for.
> >>>
> >>> At a step removed, the view server protocol itself seems like the
> >>> thing to improve on, it feels like that's the principal bottleneck.
> >>
> >> The code is here:
> >> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
> >>
> >> I’d love for someone to pick this up and give CouchDB, say, a
> ./configure
> >> --enable-native-v8 option or a plugin that allows people to opt into the
> >> speed improvements made there. :)
> >>
> >> The choice for V8 was made because of easier integration API and more
> >> reliable releases as a standalone project, which I think was a smart
> move.
> >>
> >> IIRC it relies on a change to CouchDB-y internals that has not made it
> >> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s
> reading
> >> this thread), but we should look into that and get us “native JS
> views”, at
> >> least as an option or plugin.
> >>
> >> CCing dev@.
> >>
> >> Jan
> >> --
> >>
> >>
> > Well on the first hand nifs look like a good idea but can be very
> > problematic:
> >
> > - when the view computation take time it would block the full vm
> > scheduling. It can be mitigated using a pool of threads to execute the
> work
> > asynchronously but then can create other problems like memory leaking
> etc.
> > - nifs can't be upgraded easily during hot upgrade
> > - when a nif crash, all the vm crash.
> >
> > (Note that we have the same problem when using a nif to decode/encode
> json,
> > it only works well with medium sized documents)
> >
> > One other way to improve the js handling would be removing the main
> > bottleneck ie the serialization-deserialization we do on each step. Not
> > sure if it exists but feasible, why not passing erlang terms from erlang
> > to js and js to erlang? So at the end the deserialization would happen
> only
> > on the JS side ie instead of having
> >
> > get erlang term
> > encode to json
> > send to js
> > decode json
> > process
> > encode json
> > send json
> > decode json to erlang term
> > store
> >
> > we sould just have
> >
> > get erlang term
> > send over STDIO
> > decode erlang term to JS object
> > process
> > encode to erlang term
> > send erlang term
> > store
> >
> > Erlang serialization is also very optimised.
>
> I think the ultimate goal should be to be as little
> conversion/serialisation as possible, hence no conversion to Erlang
> Terms at all.
>
> Input as string
> Parsing to get ID
> Store as string
>
> Send to JS as string
> Process with JS
> Store as string
>
> Cheers,
> Volker
>
>
>
I agree, (modulo the fact that I would replace a string by a binary ;) but
that would be only possible if we extract the metadata (_id, _rev) from the
JSON so couchdb wouldn't have to decode the JSON to get them. Streaming
json would also allows that but since there is no guaranty in the
properties order of a JSON it would be less efficient.
- benoit
Re: Erlang vs JavaScript
Posted by Volker Mische <vo...@gmail.com>.
On 08/15/2013 11:53 AM, Benoit Chesneau wrote:
> On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>> On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
>>
>>> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
>>> could have shipped an erlang view server that worked in a separate
>>> process and had the stdio overhead, to combine the slowness of the
>>> protocol with the obtuseness of erlang. ;)
>>>
>>> Evaluating Javascript within the erlang VM process intrigues me, Jens,
>>> how is that done in your case? I've not previously found the assertion
>>> that V8 would be faster than SpiderMonkey for a view server compelling
>>> since the bottleneck is almost never in the code evaluation, but I do
>>> support CouchDB switching to it for the synergy effects of a closer
>>> binding with node.js, but if it's running in the same process, that
>>> would change (though I don't immediately see why the same couldn't be
>>> done for SpiderMonkey). Off the top of my head, I don't know a safe
>>> way to evaluate JS in the VM. A NIF-based approach would either be
>>> quite elaborate or would trip all the scheduling problems that
>>> long-running NIF's are now notorious for.
>>>
>>> At a step removed, the view server protocol itself seems like the
>>> thing to improve on, it feels like that's the principal bottleneck.
>>
>> The code is here:
>> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
>>
>> I’d love for someone to pick this up and give CouchDB, say, a ./configure
>> --enable-native-v8 option or a plugin that allows people to opt into the
>> speed improvements made there. :)
>>
>> The choice for V8 was made because of easier integration API and more
>> reliable releases as a standalone project, which I think was a smart move.
>>
>> IIRC it relies on a change to CouchDB-y internals that has not made it
>> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading
>> this thread), but we should look into that and get us “native JS views”, at
>> least as an option or plugin.
>>
>> CCing dev@.
>>
>> Jan
>> --
>>
>>
> Well on the first hand nifs look like a good idea but can be very
> problematic:
>
> - when the view computation take time it would block the full vm
> scheduling. It can be mitigated using a pool of threads to execute the work
> asynchronously but then can create other problems like memory leaking etc.
> - nifs can't be upgraded easily during hot upgrade
> - when a nif crash, all the vm crash.
>
> (Note that we have the same problem when using a nif to decode/encode json,
> it only works well with medium sized documents)
>
> One other way to improve the js handling would be removing the main
> bottleneck ie the serialization-deserialization we do on each step. Not
> sure if it exists but feasible, why not passing erlang terms from erlang
> to js and js to erlang? So at the end the deserialization would happen only
> on the JS side ie instead of having
>
> get erlang term
> encode to json
> send to js
> decode json
> process
> encode json
> send json
> decode json to erlang term
> store
>
> we sould just have
>
> get erlang term
> send over STDIO
> decode erlang term to JS object
> process
> encode to erlang term
> send erlang term
> store
>
> Erlang serialization is also very optimised.
I think the ultimate goal should be to be as little
conversion/serialisation as possible, hence no conversion to Erlang
Terms at all.
Input as string
Parsing to get ID
Store as string
Send to JS as string
Process with JS
Store as string
Cheers,
Volker
Re: Erlang vs JavaScript
Posted by Volker Mische <vo...@gmail.com>.
On 08/15/2013 11:53 AM, Benoit Chesneau wrote:
> On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>> On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
>>
>>> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
>>> could have shipped an erlang view server that worked in a separate
>>> process and had the stdio overhead, to combine the slowness of the
>>> protocol with the obtuseness of erlang. ;)
>>>
>>> Evaluating Javascript within the erlang VM process intrigues me, Jens,
>>> how is that done in your case? I've not previously found the assertion
>>> that V8 would be faster than SpiderMonkey for a view server compelling
>>> since the bottleneck is almost never in the code evaluation, but I do
>>> support CouchDB switching to it for the synergy effects of a closer
>>> binding with node.js, but if it's running in the same process, that
>>> would change (though I don't immediately see why the same couldn't be
>>> done for SpiderMonkey). Off the top of my head, I don't know a safe
>>> way to evaluate JS in the VM. A NIF-based approach would either be
>>> quite elaborate or would trip all the scheduling problems that
>>> long-running NIF's are now notorious for.
>>>
>>> At a step removed, the view server protocol itself seems like the
>>> thing to improve on, it feels like that's the principal bottleneck.
>>
>> The code is here:
>> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
>>
>> I’d love for someone to pick this up and give CouchDB, say, a ./configure
>> --enable-native-v8 option or a plugin that allows people to opt into the
>> speed improvements made there. :)
>>
>> The choice for V8 was made because of easier integration API and more
>> reliable releases as a standalone project, which I think was a smart move.
>>
>> IIRC it relies on a change to CouchDB-y internals that has not made it
>> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading
>> this thread), but we should look into that and get us “native JS views”, at
>> least as an option or plugin.
>>
>> CCing dev@.
>>
>> Jan
>> --
>>
>>
> Well on the first hand nifs look like a good idea but can be very
> problematic:
>
> - when the view computation take time it would block the full vm
> scheduling. It can be mitigated using a pool of threads to execute the work
> asynchronously but then can create other problems like memory leaking etc.
> - nifs can't be upgraded easily during hot upgrade
> - when a nif crash, all the vm crash.
>
> (Note that we have the same problem when using a nif to decode/encode json,
> it only works well with medium sized documents)
>
> One other way to improve the js handling would be removing the main
> bottleneck ie the serialization-deserialization we do on each step. Not
> sure if it exists but feasible, why not passing erlang terms from erlang
> to js and js to erlang? So at the end the deserialization would happen only
> on the JS side ie instead of having
>
> get erlang term
> encode to json
> send to js
> decode json
> process
> encode json
> send json
> decode json to erlang term
> store
>
> we sould just have
>
> get erlang term
> send over STDIO
> decode erlang term to JS object
> process
> encode to erlang term
> send erlang term
> store
>
> Erlang serialization is also very optimised.
I think the ultimate goal should be to be as little
conversion/serialisation as possible, hence no conversion to Erlang
Terms at all.
Input as string
Parsing to get ID
Store as string
Send to JS as string
Process with JS
Store as string
Cheers,
Volker
Re: Erlang vs JavaScript
Posted by Jan Lehnardt <ja...@apache.org>.
On Aug 15, 2013, at 11:53 , Benoit Chesneau <bc...@gmail.com> wrote:
> On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>> On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
>>
>>> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
>>> could have shipped an erlang view server that worked in a separate
>>> process and had the stdio overhead, to combine the slowness of the
>>> protocol with the obtuseness of erlang. ;)
>>>
>>> Evaluating Javascript within the erlang VM process intrigues me, Jens,
>>> how is that done in your case? I've not previously found the assertion
>>> that V8 would be faster than SpiderMonkey for a view server compelling
>>> since the bottleneck is almost never in the code evaluation, but I do
>>> support CouchDB switching to it for the synergy effects of a closer
>>> binding with node.js, but if it's running in the same process, that
>>> would change (though I don't immediately see why the same couldn't be
>>> done for SpiderMonkey). Off the top of my head, I don't know a safe
>>> way to evaluate JS in the VM. A NIF-based approach would either be
>>> quite elaborate or would trip all the scheduling problems that
>>> long-running NIF's are now notorious for.
>>>
>>> At a step removed, the view server protocol itself seems like the
>>> thing to improve on, it feels like that's the principal bottleneck.
>>
>> The code is here:
>> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
>>
>> I’d love for someone to pick this up and give CouchDB, say, a ./configure
>> --enable-native-v8 option or a plugin that allows people to opt into the
>> speed improvements made there. :)
>>
>> The choice for V8 was made because of easier integration API and more
>> reliable releases as a standalone project, which I think was a smart move.
>>
>> IIRC it relies on a change to CouchDB-y internals that has not made it
>> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading
>> this thread), but we should look into that and get us “native JS views”, at
>> least as an option or plugin.
>>
>> CCing dev@.
>>
>> Jan
>> --
>>
>>
> Well on the first hand nifs look like a good idea but can be very
> problematic:
>
> - when the view computation take time it would block the full vm
> scheduling. It can be mitigated using a pool of threads to execute the work
> asynchronously but then can create other problems like memory leaking etc.
> - nifs can't be upgraded easily during hot upgrade
> - when a nif crash, all the vm crash.
Yeah totally, hence making the whole thing an option.
> (Note that we have the same problem when using a nif to decode/encode json,
> it only works well with medium sized documents)
> One other way to improve the js handling would be removing the main
> bottleneck ie the serialization-deserialization we do on each step. Not
> sure if it exists but feasible, why not passing erlang terms from erlang
> to js and js to erlang? So at the end the deserialization would happen only
> on the JS side ie instead of having
>
> get erlang term
> encode to json
> send to js
> decode json
> process
> encode json
> send json
> decode json to erlang term
> store
>
> we sould just have
>
> get erlang term
> send over STDIO
> decode erlang term to JS object
> process
> encode to erlang term
> send erlang term
> store
>
> Erlang serialization is also very optimised.
>
>
> Both solutions could co-exist, that may worh a try and benchmark each...
I think we just want both solutions period, the embedded one will still be faster, but potentially a little less stable, and the external view server one will be slower but extremely robust. Users should be able to choose between them :)
Best
Jan
--
Re: Erlang vs JavaScript
Posted by Jan Lehnardt <ja...@apache.org>.
On Aug 15, 2013, at 11:53 , Benoit Chesneau <bc...@gmail.com> wrote:
> On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>> On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
>>
>>> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
>>> could have shipped an erlang view server that worked in a separate
>>> process and had the stdio overhead, to combine the slowness of the
>>> protocol with the obtuseness of erlang. ;)
>>>
>>> Evaluating Javascript within the erlang VM process intrigues me, Jens,
>>> how is that done in your case? I've not previously found the assertion
>>> that V8 would be faster than SpiderMonkey for a view server compelling
>>> since the bottleneck is almost never in the code evaluation, but I do
>>> support CouchDB switching to it for the synergy effects of a closer
>>> binding with node.js, but if it's running in the same process, that
>>> would change (though I don't immediately see why the same couldn't be
>>> done for SpiderMonkey). Off the top of my head, I don't know a safe
>>> way to evaluate JS in the VM. A NIF-based approach would either be
>>> quite elaborate or would trip all the scheduling problems that
>>> long-running NIF's are now notorious for.
>>>
>>> At a step removed, the view server protocol itself seems like the
>>> thing to improve on, it feels like that's the principal bottleneck.
>>
>> The code is here:
>> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
>>
>> I’d love for someone to pick this up and give CouchDB, say, a ./configure
>> --enable-native-v8 option or a plugin that allows people to opt into the
>> speed improvements made there. :)
>>
>> The choice for V8 was made because of easier integration API and more
>> reliable releases as a standalone project, which I think was a smart move.
>>
>> IIRC it relies on a change to CouchDB-y internals that has not made it
>> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading
>> this thread), but we should look into that and get us “native JS views”, at
>> least as an option or plugin.
>>
>> CCing dev@.
>>
>> Jan
>> --
>>
>>
> Well on the first hand nifs look like a good idea but can be very
> problematic:
>
> - when the view computation take time it would block the full vm
> scheduling. It can be mitigated using a pool of threads to execute the work
> asynchronously but then can create other problems like memory leaking etc.
> - nifs can't be upgraded easily during hot upgrade
> - when a nif crash, all the vm crash.
Yeah totally, hence making the whole thing an option.
> (Note that we have the same problem when using a nif to decode/encode json,
> it only works well with medium sized documents)
> One other way to improve the js handling would be removing the main
> bottleneck ie the serialization-deserialization we do on each step. Not
> sure if it exists but feasible, why not passing erlang terms from erlang
> to js and js to erlang? So at the end the deserialization would happen only
> on the JS side ie instead of having
>
> get erlang term
> encode to json
> send to js
> decode json
> process
> encode json
> send json
> decode json to erlang term
> store
>
> we sould just have
>
> get erlang term
> send over STDIO
> decode erlang term to JS object
> process
> encode to erlang term
> send erlang term
> store
>
> Erlang serialization is also very optimised.
>
>
> Both solutions could co-exist, that may worh a try and benchmark each...
I think we just want both solutions period, the embedded one will still be faster, but potentially a little less stable, and the external view server one will be slower but extremely robust. Users should be able to choose between them :)
Best
Jan
--
Re: Erlang vs JavaScript
Posted by Benoit Chesneau <bc...@gmail.com>.
On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
>
> > A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
> > could have shipped an erlang view server that worked in a separate
> > process and had the stdio overhead, to combine the slowness of the
> > protocol with the obtuseness of erlang. ;)
> >
> > Evaluating Javascript within the erlang VM process intrigues me, Jens,
> > how is that done in your case? I've not previously found the assertion
> > that V8 would be faster than SpiderMonkey for a view server compelling
> > since the bottleneck is almost never in the code evaluation, but I do
> > support CouchDB switching to it for the synergy effects of a closer
> > binding with node.js, but if it's running in the same process, that
> > would change (though I don't immediately see why the same couldn't be
> > done for SpiderMonkey). Off the top of my head, I don't know a safe
> > way to evaluate JS in the VM. A NIF-based approach would either be
> > quite elaborate or would trip all the scheduling problems that
> > long-running NIF's are now notorious for.
> >
> > At a step removed, the view server protocol itself seems like the
> > thing to improve on, it feels like that's the principal bottleneck.
>
> The code is here:
> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
>
> I’d love for someone to pick this up and give CouchDB, say, a ./configure
> --enable-native-v8 option or a plugin that allows people to opt into the
> speed improvements made there. :)
>
> The choice for V8 was made because of easier integration API and more
> reliable releases as a standalone project, which I think was a smart move.
>
> IIRC it relies on a change to CouchDB-y internals that has not made it
> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading
> this thread), but we should look into that and get us “native JS views”, at
> least as an option or plugin.
>
> CCing dev@.
>
> Jan
> --
>
>
Well on the first hand nifs look like a good idea but can be very
problematic:
- when the view computation take time it would block the full vm
scheduling. It can be mitigated using a pool of threads to execute the work
asynchronously but then can create other problems like memory leaking etc.
- nifs can't be upgraded easily during hot upgrade
- when a nif crash, all the vm crash.
(Note that we have the same problem when using a nif to decode/encode json,
it only works well with medium sized documents)
One other way to improve the js handling would be removing the main
bottleneck ie the serialization-deserialization we do on each step. Not
sure if it exists but feasible, why not passing erlang terms from erlang
to js and js to erlang? So at the end the deserialization would happen only
on the JS side ie instead of having
get erlang term
encode to json
send to js
decode json
process
encode json
send json
decode json to erlang term
store
we sould just have
get erlang term
send over STDIO
decode erlang term to JS object
process
encode to erlang term
send erlang term
store
Erlang serialization is also very optimised.
Both solutions could co-exist, that may worh a try and benchmark each...
- benoit
Re: Erlang vs JavaScript
Posted by Benoit Chesneau <bc...@gmail.com>.
On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
>
> > A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
> > could have shipped an erlang view server that worked in a separate
> > process and had the stdio overhead, to combine the slowness of the
> > protocol with the obtuseness of erlang. ;)
> >
> > Evaluating Javascript within the erlang VM process intrigues me, Jens,
> > how is that done in your case? I've not previously found the assertion
> > that V8 would be faster than SpiderMonkey for a view server compelling
> > since the bottleneck is almost never in the code evaluation, but I do
> > support CouchDB switching to it for the synergy effects of a closer
> > binding with node.js, but if it's running in the same process, that
> > would change (though I don't immediately see why the same couldn't be
> > done for SpiderMonkey). Off the top of my head, I don't know a safe
> > way to evaluate JS in the VM. A NIF-based approach would either be
> > quite elaborate or would trip all the scheduling problems that
> > long-running NIF's are now notorious for.
> >
> > At a step removed, the view server protocol itself seems like the
> > thing to improve on, it feels like that's the principal bottleneck.
>
> The code is here:
> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
>
> I’d love for someone to pick this up and give CouchDB, say, a ./configure
> --enable-native-v8 option or a plugin that allows people to opt into the
> speed improvements made there. :)
>
> The choice for V8 was made because of easier integration API and more
> reliable releases as a standalone project, which I think was a smart move.
>
> IIRC it relies on a change to CouchDB-y internals that has not made it
> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading
> this thread), but we should look into that and get us “native JS views”, at
> least as an option or plugin.
>
> CCing dev@.
>
> Jan
> --
>
>
Well on the first hand nifs look like a good idea but can be very
problematic:
- when the view computation take time it would block the full vm
scheduling. It can be mitigated using a pool of threads to execute the work
asynchronously but then can create other problems like memory leaking etc.
- nifs can't be upgraded easily during hot upgrade
- when a nif crash, all the vm crash.
(Note that we have the same problem when using a nif to decode/encode json,
it only works well with medium sized documents)
One other way to improve the js handling would be removing the main
bottleneck ie the serialization-deserialization we do on each step. Not
sure if it exists but feasible, why not passing erlang terms from erlang
to js and js to erlang? So at the end the deserialization would happen only
on the JS side ie instead of having
get erlang term
encode to json
send to js
decode json
process
encode json
send json
decode json to erlang term
store
we sould just have
get erlang term
send over STDIO
decode erlang term to JS object
process
encode to erlang term
send erlang term
store
Erlang serialization is also very optimised.
Both solutions could co-exist, that may worh a try and benchmark each...
- benoit
Re: Erlang vs JavaScript
Posted by Jan Lehnardt <ja...@apache.org>.
On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
> could have shipped an erlang view server that worked in a separate
> process and had the stdio overhead, to combine the slowness of the
> protocol with the obtuseness of erlang. ;)
>
> Evaluating Javascript within the erlang VM process intrigues me, Jens,
> how is that done in your case? I've not previously found the assertion
> that V8 would be faster than SpiderMonkey for a view server compelling
> since the bottleneck is almost never in the code evaluation, but I do
> support CouchDB switching to it for the synergy effects of a closer
> binding with node.js, but if it's running in the same process, that
> would change (though I don't immediately see why the same couldn't be
> done for SpiderMonkey). Off the top of my head, I don't know a safe
> way to evaluate JS in the VM. A NIF-based approach would either be
> quite elaborate or would trip all the scheduling problems that
> long-running NIF's are now notorious for.
>
> At a step removed, the view server protocol itself seems like the
> thing to improve on, it feels like that's the principal bottleneck.
The code is here: https://github.com/couchbase/couchdb/tree/master/src/mapreduce
I’d love for someone to pick this up and give CouchDB, say, a ./configure --enable-native-v8 option or a plugin that allows people to opt into the speed improvements made there. :)
The choice for V8 was made because of easier integration API and more reliable releases as a standalone project, which I think was a smart move.
IIRC it relies on a change to CouchDB-y internals that has not made it back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading this thread), but we should look into that and get us “native JS views”, at least as an option or plugin.
CCing dev@.
Jan
--
>
> B.
>
>
> On 15 August 2013 08:22, Jason Smith <jh...@apache.org> wrote:
>> Yes, to a first approximation, with a native view, CouchDB is basically
>> running eval() on your code. In my example, I took advantage of this to
>> build a nonstandard response to satisfy an application. (Instead of a 404,
>> we sent a designated fallback document body.)
>>
>> But, if you accumulate the list in a native view, a JavaScript view, or a
>> hypothetical Erlang view (i.e. a subprocess), from the operating system's
>> perspective, the memory for that list will be allocated somewhere. Either
>> the CouchDB process asks for X KB more memory, or its subprocess will ask
>> for it. So I think the total system impact is probably low in practice.
>>
>> So I guess my point is not that native views are wrong, just they have a
>> cost so you should weigh the cost/benefit for your own project. In the case
>> of manage_couchdb, I wrote a JavaScript implementation; but since sometimes
>> I have an emergency and I must find conflicts ASAP, I made an Erlang
>> version because it is worth it.
>>
>>
>> On Thu, Aug 15, 2013 at 2:05 PM, Stanley Iriele <si...@gmail.com>wrote:
>>
>>> Whoa...OK...that I had no idea about...thanks for taking the time to go to
>>> that granularity, by the way.
>>>
>>> So does this mean that the process memory is shared? As apposed to living
>>> in its own space?.so if someone accumulates a large json object in a list
>>> function its chewing up couchdb's memory?... I guess I'm a little confused
>>> about what's in the same process and what isn't now
>>> On Aug 14, 2013 11:57 PM, "Jason Smith" <jh...@apache.org> wrote:
>>>
>>>> To me, an Erlang view is a view server which supports map, reduce, show,
>>>> update, list, etc. functions in the Erlang language. (Basically it is
>>>> implemented in Erlang.)
>>>>
>>>> A view server is a subprocess that runs beneath CouchDB which
>>> communicates
>>>> with it over standard i/o. It is a different process in the operating
>>>> system and only interfaces with the main server using the view server
>>>> protocol (basically a bunch of JSON messages going back and forth).
>>>>
>>>> I do not know of an Erlang view server which works well and is currently
>>>> maintained.
>>>>
>>>> A native view (shipped by CouchDB but disabled by default) is some
>>>> corner-cutting. Code is evaluated directly by the primary CouchDB server.
>>>> Since CouchDB is Erlang, the native query server is necessarily Erlang.
>>> The
>>>> key difference is, your code is right there in the eye of the storm. You
>>>> can call couch_server:open("some_db") and completely circumvent security
>>>> and other invariants which CouchDB enforces. You can leak memory until
>>> the
>>>> kernel OOM killer terminates CouchDB. It's not about the language, it's
>>>> that is is running inside the CouchDB process.
>>>>
>>>>
>>>>
>>>> On Thu, Aug 15, 2013 at 1:36 PM, Stanley Iriele <siriele2x3@gmail.com
>>>>> wrote:
>>>>
>>>>> Wait....I'm a tad confused here..Jason what is the difference between
>>>>> native views and Erlang views?...
>>>>> On Aug 14, 2013 11:16 PM, "Jason Smith" <jh...@apache.org> wrote:
>>>>>
>>>>>> Oh, also:
>>>>>>
>>>>>> They are **not** Erlang views. They are **native** views. We should
>>>>>> emphasize the latter to remind ourselves about the security and
>>>>> reliability
>>>>>> risks which Bob identifies.
>>>>>>
>>>>>> They are very powerful, but it is a trade-off. Once I had a customer
>>>> who
>>>>>> had a basic "class" document describing common values. All other
>>>>> documents
>>>>>> were for modifications to the "base class" so to speak. He needed to
>>>>> query
>>>>>> by document ID, but if no such document existed, return the "base
>>>> class"
>>>>>> document instead. The product was already in the field and so the
>>> code
>>>>>> could not change. We had to change it in CouchDB.
>>>>>>
>>>>>> The fix was very simple: a _rewrite rule to a native _show function.
>>> In
>>>>> the
>>>>>> show function, if the Doc was null, then we used the internal CouchDB
>>>> API
>>>>>> to fetch the default document. Voila.
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 15, 2013 at 1:08 PM, Jason Smith <jh...@apache.org> wrote:
>>>>>>
>>>>>>> On Thursday, August 15, 2013, Andrey Kuprianov wrote:
>>>>>>>
>>>>>>>> Doesnt server performance downgrade, while views are being
>>> rebuilt?
>>>> So
>>>>>> the
>>>>>>>> faster they are rebuilt, the better for you.
>>>>>>>
>>>>>>>
>>>>>>> If my view build would degrade total performance to cross an
>>>>> unacceptable
>>>>>>> threshold, then I am really riding the line! What about an
>>> unplanned
>>>>>>> compaction? What if one day the clients have a bug and load
>>>> increases?
>>>>>> What
>>>>>>> if an unplanned disaster happens and a backup must be performed
>>>>> urgently?
>>>>>>>
>>>>>>> I would evaluate view performance in the larger context of the
>>> entire
>>>>>>> application life cycle.
>>>>>>>
>>>>>>> Men seem to want to date beautiful women. It is a very high
>>> priority
>>>> at
>>>>>>> the pub or whatever. But long-married men do not even think about
>>>> their
>>>>>>> wife's attractiveness because that is a small, superficial part of
>>> a
>>>>> much
>>>>>>> larger story.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Besides, looks like it's possible to do the same 3 steps with
>>> design
>>>>> doc
>>>>>>>> views created in Erlang? Or is it just about using require() in
>>>>> Node.js?
>>>>>>>>
>>>>>>>
>>>>>>> Actually, yes that is a fine point. I myself prefer Node.js but
>>>> anyone
>>>>>> can
>>>>>>> choose the best fit for them.
>>>>>>>
>>>>>>> And speaking more broadly, CouchDB is a very flexible platform so
>>> it
>>>> is
>>>>>>> quite likely that my own policies do not apply to every use case.
>>> In
>>>>> fact
>>>>>>> if I'm honest I use native views myself, usually for unplanned
>>>>>>> troubleshooting, I want to find conflicts so I use manage_couchdb:
>>>>>>> http://github.com/iriscouch/manage_couchdb
>>>>>>>
>>>>>>> My main point is, anybody time somebody says "performance" ask
>>>> yourself
>>>>>> if
>>>>>>> it is really a "performance siren." Earlier in this thread, Jens
>>>> raises
>>>>>>> some examples of plausible true performance requirements, not just
>>>>> siren
>>>>>>> songs.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
Re: Erlang vs JavaScript
Posted by Jan Lehnardt <ja...@apache.org>.
On Aug 15, 2013, at 10:09 , Robert Newson <rn...@apache.org> wrote:
> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
> could have shipped an erlang view server that worked in a separate
> process and had the stdio overhead, to combine the slowness of the
> protocol with the obtuseness of erlang. ;)
>
> Evaluating Javascript within the erlang VM process intrigues me, Jens,
> how is that done in your case? I've not previously found the assertion
> that V8 would be faster than SpiderMonkey for a view server compelling
> since the bottleneck is almost never in the code evaluation, but I do
> support CouchDB switching to it for the synergy effects of a closer
> binding with node.js, but if it's running in the same process, that
> would change (though I don't immediately see why the same couldn't be
> done for SpiderMonkey). Off the top of my head, I don't know a safe
> way to evaluate JS in the VM. A NIF-based approach would either be
> quite elaborate or would trip all the scheduling problems that
> long-running NIF's are now notorious for.
>
> At a step removed, the view server protocol itself seems like the
> thing to improve on, it feels like that's the principal bottleneck.
The code is here: https://github.com/couchbase/couchdb/tree/master/src/mapreduce
I’d love for someone to pick this up and give CouchDB, say, a ./configure --enable-native-v8 option or a plugin that allows people to opt into the speed improvements made there. :)
The choice for V8 was made because of easier integration API and more reliable releases as a standalone project, which I think was a smart move.
IIRC it relies on a change to CouchDB-y internals that has not made it back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading this thread), but we should look into that and get us “native JS views”, at least as an option or plugin.
CCing dev@.
Jan
--
>
> B.
>
>
> On 15 August 2013 08:22, Jason Smith <jh...@apache.org> wrote:
>> Yes, to a first approximation, with a native view, CouchDB is basically
>> running eval() on your code. In my example, I took advantage of this to
>> build a nonstandard response to satisfy an application. (Instead of a 404,
>> we sent a designated fallback document body.)
>>
>> But, if you accumulate the list in a native view, a JavaScript view, or a
>> hypothetical Erlang view (i.e. a subprocess), from the operating system's
>> perspective, the memory for that list will be allocated somewhere. Either
>> the CouchDB process asks for X KB more memory, or its subprocess will ask
>> for it. So I think the total system impact is probably low in practice.
>>
>> So I guess my point is not that native views are wrong, just they have a
>> cost so you should weigh the cost/benefit for your own project. In the case
>> of manage_couchdb, I wrote a JavaScript implementation; but since sometimes
>> I have an emergency and I must find conflicts ASAP, I made an Erlang
>> version because it is worth it.
>>
>>
>> On Thu, Aug 15, 2013 at 2:05 PM, Stanley Iriele <si...@gmail.com>wrote:
>>
>>> Whoa...OK...that I had no idea about...thanks for taking the time to go to
>>> that granularity, by the way.
>>>
>>> So does this mean that the process memory is shared? As apposed to living
>>> in its own space?.so if someone accumulates a large json object in a list
>>> function its chewing up couchdb's memory?... I guess I'm a little confused
>>> about what's in the same process and what isn't now
>>> On Aug 14, 2013 11:57 PM, "Jason Smith" <jh...@apache.org> wrote:
>>>
>>>> To me, an Erlang view is a view server which supports map, reduce, show,
>>>> update, list, etc. functions in the Erlang language. (Basically it is
>>>> implemented in Erlang.)
>>>>
>>>> A view server is a subprocess that runs beneath CouchDB which
>>> communicates
>>>> with it over standard i/o. It is a different process in the operating
>>>> system and only interfaces with the main server using the view server
>>>> protocol (basically a bunch of JSON messages going back and forth).
>>>>
>>>> I do not know of an Erlang view server which works well and is currently
>>>> maintained.
>>>>
>>>> A native view (shipped by CouchDB but disabled by default) is some
>>>> corner-cutting. Code is evaluated directly by the primary CouchDB server.
>>>> Since CouchDB is Erlang, the native query server is necessarily Erlang.
>>> The
>>>> key difference is, your code is right there in the eye of the storm. You
>>>> can call couch_server:open("some_db") and completely circumvent security
>>>> and other invariants which CouchDB enforces. You can leak memory until
>>> the
>>>> kernel OOM killer terminates CouchDB. It's not about the language, it's
>>>> that is is running inside the CouchDB process.
>>>>
>>>>
>>>>
>>>> On Thu, Aug 15, 2013 at 1:36 PM, Stanley Iriele <siriele2x3@gmail.com
>>>>> wrote:
>>>>
>>>>> Wait....I'm a tad confused here..Jason what is the difference between
>>>>> native views and Erlang views?...
>>>>> On Aug 14, 2013 11:16 PM, "Jason Smith" <jh...@apache.org> wrote:
>>>>>
>>>>>> Oh, also:
>>>>>>
>>>>>> They are **not** Erlang views. They are **native** views. We should
>>>>>> emphasize the latter to remind ourselves about the security and
>>>>> reliability
>>>>>> risks which Bob identifies.
>>>>>>
>>>>>> They are very powerful, but it is a trade-off. Once I had a customer
>>>> who
>>>>>> had a basic "class" document describing common values. All other
>>>>> documents
>>>>>> were for modifications to the "base class" so to speak. He needed to
>>>>> query
>>>>>> by document ID, but if no such document existed, return the "base
>>>> class"
>>>>>> document instead. The product was already in the field and so the
>>> code
>>>>>> could not change. We had to change it in CouchDB.
>>>>>>
>>>>>> The fix was very simple: a _rewrite rule to a native _show function.
>>> In
>>>>> the
>>>>>> show function, if the Doc was null, then we used the internal CouchDB
>>>> API
>>>>>> to fetch the default document. Voila.
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 15, 2013 at 1:08 PM, Jason Smith <jh...@apache.org> wrote:
>>>>>>
>>>>>>> On Thursday, August 15, 2013, Andrey Kuprianov wrote:
>>>>>>>
>>>>>>>> Doesnt server performance downgrade, while views are being
>>> rebuilt?
>>>> So
>>>>>> the
>>>>>>>> faster they are rebuilt, the better for you.
>>>>>>>
>>>>>>>
>>>>>>> If my view build would degrade total performance to cross an
>>>>> unacceptable
>>>>>>> threshold, then I am really riding the line! What about an
>>> unplanned
>>>>>>> compaction? What if one day the clients have a bug and load
>>>> increases?
>>>>>> What
>>>>>>> if an unplanned disaster happens and a backup must be performed
>>>>> urgently?
>>>>>>>
>>>>>>> I would evaluate view performance in the larger context of the
>>> entire
>>>>>>> application life cycle.
>>>>>>>
>>>>>>> Men seem to want to date beautiful women. It is a very high
>>> priority
>>>> at
>>>>>>> the pub or whatever. But long-married men do not even think about
>>>> their
>>>>>>> wife's attractiveness because that is a small, superficial part of
>>> a
>>>>> much
>>>>>>> larger story.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Besides, looks like it's possible to do the same 3 steps with
>>> design
>>>>> doc
>>>>>>>> views created in Erlang? Or is it just about using require() in
>>>>> Node.js?
>>>>>>>>
>>>>>>>
>>>>>>> Actually, yes that is a fine point. I myself prefer Node.js but
>>>> anyone
>>>>>> can
>>>>>>> choose the best fit for them.
>>>>>>>
>>>>>>> And speaking more broadly, CouchDB is a very flexible platform so
>>> it
>>>> is
>>>>>>> quite likely that my own policies do not apply to every use case.
>>> In
>>>>> fact
>>>>>>> if I'm honest I use native views myself, usually for unplanned
>>>>>>> troubleshooting, I want to find conflicts so I use manage_couchdb:
>>>>>>> http://github.com/iriscouch/manage_couchdb
>>>>>>>
>>>>>>> My main point is, anybody time somebody says "performance" ask
>>>> yourself
>>>>>> if
>>>>>>> it is really a "performance siren." Earlier in this thread, Jens
>>>> raises
>>>>>>> some examples of plausible true performance requirements, not just
>>>>> siren
>>>>>>> songs.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
Re: Erlang vs JavaScript
Posted by Jens Alfke <je...@couchbase.com>.
On Aug 15, 2013, at 1:09 AM, Robert Newson <rn...@apache.org> wrote:
> Off the top of my head, I don't know a safe
> way to evaluate JS in the VM. A NIF-based approach would either be
> quite elaborate or would trip all the scheduling problems that
> long-running NIF's are now notorious for.
Yeah, there could be problems with this. For example, a couple of the standard CouchDB unit tests failed — IIRC there is one that intentionally blows up the JS stack with an infinite regress, which eventually causes SpiderMonkey to segfault. I could see that being a problem for hosted environments where multiple developers are sharing a single CouchDB instance.
—Jens
Re: Erlang vs JavaScript
Posted by Robert Newson <rn...@apache.org>.
A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
could have shipped an erlang view server that worked in a separate
process and had the stdio overhead, to combine the slowness of the
protocol with the obtuseness of erlang. ;)
Evaluating Javascript within the erlang VM process intrigues me, Jens,
how is that done in your case? I've not previously found the assertion
that V8 would be faster than SpiderMonkey for a view server compelling
since the bottleneck is almost never in the code evaluation, but I do
support CouchDB switching to it for the synergy effects of a closer
binding with node.js, but if it's running in the same process, that
would change (though I don't immediately see why the same couldn't be
done for SpiderMonkey). Off the top of my head, I don't know a safe
way to evaluate JS in the VM. A NIF-based approach would either be
quite elaborate or would trip all the scheduling problems that
long-running NIF's are now notorious for.
At a step removed, the view server protocol itself seems like the
thing to improve on, it feels like that's the principal bottleneck.
B.
On 15 August 2013 08:22, Jason Smith <jh...@apache.org> wrote:
> Yes, to a first approximation, with a native view, CouchDB is basically
> running eval() on your code. In my example, I took advantage of this to
> build a nonstandard response to satisfy an application. (Instead of a 404,
> we sent a designated fallback document body.)
>
> But, if you accumulate the list in a native view, a JavaScript view, or a
> hypothetical Erlang view (i.e. a subprocess), from the operating system's
> perspective, the memory for that list will be allocated somewhere. Either
> the CouchDB process asks for X KB more memory, or its subprocess will ask
> for it. So I think the total system impact is probably low in practice.
>
> So I guess my point is not that native views are wrong, just they have a
> cost so you should weigh the cost/benefit for your own project. In the case
> of manage_couchdb, I wrote a JavaScript implementation; but since sometimes
> I have an emergency and I must find conflicts ASAP, I made an Erlang
> version because it is worth it.
>
>
> On Thu, Aug 15, 2013 at 2:05 PM, Stanley Iriele <si...@gmail.com>wrote:
>
>> Whoa...OK...that I had no idea about...thanks for taking the time to go to
>> that granularity, by the way.
>>
>> So does this mean that the process memory is shared? As apposed to living
>> in its own space?.so if someone accumulates a large json object in a list
>> function its chewing up couchdb's memory?... I guess I'm a little confused
>> about what's in the same process and what isn't now
>> On Aug 14, 2013 11:57 PM, "Jason Smith" <jh...@apache.org> wrote:
>>
>> > To me, an Erlang view is a view server which supports map, reduce, show,
>> > update, list, etc. functions in the Erlang language. (Basically it is
>> > implemented in Erlang.)
>> >
>> > A view server is a subprocess that runs beneath CouchDB which
>> communicates
>> > with it over standard i/o. It is a different process in the operating
>> > system and only interfaces with the main server using the view server
>> > protocol (basically a bunch of JSON messages going back and forth).
>> >
>> > I do not know of an Erlang view server which works well and is currently
>> > maintained.
>> >
>> > A native view (shipped by CouchDB but disabled by default) is some
>> > corner-cutting. Code is evaluated directly by the primary CouchDB server.
>> > Since CouchDB is Erlang, the native query server is necessarily Erlang.
>> The
>> > key difference is, your code is right there in the eye of the storm. You
>> > can call couch_server:open("some_db") and completely circumvent security
>> > and other invariants which CouchDB enforces. You can leak memory until
>> the
>> > kernel OOM killer terminates CouchDB. It's not about the language, it's
>> > that is is running inside the CouchDB process.
>> >
>> >
>> >
>> > On Thu, Aug 15, 2013 at 1:36 PM, Stanley Iriele <siriele2x3@gmail.com
>> > >wrote:
>> >
>> > > Wait....I'm a tad confused here..Jason what is the difference between
>> > > native views and Erlang views?...
>> > > On Aug 14, 2013 11:16 PM, "Jason Smith" <jh...@apache.org> wrote:
>> > >
>> > > > Oh, also:
>> > > >
>> > > > They are **not** Erlang views. They are **native** views. We should
>> > > > emphasize the latter to remind ourselves about the security and
>> > > reliability
>> > > > risks which Bob identifies.
>> > > >
>> > > > They are very powerful, but it is a trade-off. Once I had a customer
>> > who
>> > > > had a basic "class" document describing common values. All other
>> > > documents
>> > > > were for modifications to the "base class" so to speak. He needed to
>> > > query
>> > > > by document ID, but if no such document existed, return the "base
>> > class"
>> > > > document instead. The product was already in the field and so the
>> code
>> > > > could not change. We had to change it in CouchDB.
>> > > >
>> > > > The fix was very simple: a _rewrite rule to a native _show function.
>> In
>> > > the
>> > > > show function, if the Doc was null, then we used the internal CouchDB
>> > API
>> > > > to fetch the default document. Voila.
>> > > >
>> > > >
>> > > > On Thu, Aug 15, 2013 at 1:08 PM, Jason Smith <jh...@apache.org> wrote:
>> > > >
>> > > > > On Thursday, August 15, 2013, Andrey Kuprianov wrote:
>> > > > >
>> > > > >> Doesnt server performance downgrade, while views are being
>> rebuilt?
>> > So
>> > > > the
>> > > > >> faster they are rebuilt, the better for you.
>> > > > >
>> > > > >
>> > > > > If my view build would degrade total performance to cross an
>> > > unacceptable
>> > > > > threshold, then I am really riding the line! What about an
>> unplanned
>> > > > > compaction? What if one day the clients have a bug and load
>> > increases?
>> > > > What
>> > > > > if an unplanned disaster happens and a backup must be performed
>> > > urgently?
>> > > > >
>> > > > > I would evaluate view performance in the larger context of the
>> entire
>> > > > > application life cycle.
>> > > > >
>> > > > > Men seem to want to date beautiful women. It is a very high
>> priority
>> > at
>> > > > > the pub or whatever. But long-married men do not even think about
>> > their
>> > > > > wife's attractiveness because that is a small, superficial part of
>> a
>> > > much
>> > > > > larger story.
>> > > > >
>> > > > >
>> > > > >>
>> > > > >> Besides, looks like it's possible to do the same 3 steps with
>> design
>> > > doc
>> > > > >> views created in Erlang? Or is it just about using require() in
>> > > Node.js?
>> > > > >>
>> > > > >
>> > > > > Actually, yes that is a fine point. I myself prefer Node.js but
>> > anyone
>> > > > can
>> > > > > choose the best fit for them.
>> > > > >
>> > > > > And speaking more broadly, CouchDB is a very flexible platform so
>> it
>> > is
>> > > > > quite likely that my own policies do not apply to every use case.
>> In
>> > > fact
>> > > > > if I'm honest I use native views myself, usually for unplanned
>> > > > > troubleshooting, I want to find conflicts so I use manage_couchdb:
>> > > > > http://github.com/iriscouch/manage_couchdb
>> > > > >
>> > > > > My main point is, anybody time somebody says "performance" ask
>> > yourself
>> > > > if
>> > > > > it is really a "performance siren." Earlier in this thread, Jens
>> > raises
>> > > > > some examples of plausible true performance requirements, not just
>> > > siren
>> > > > > songs.
>> > > > >
>> > > >
>> > >
>> >
>>
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
Yes, to a first approximation, with a native view, CouchDB is basically
running eval() on your code. In my example, I took advantage of this to
build a nonstandard response to satisfy an application. (Instead of a 404,
we sent a designated fallback document body.)
But, if you accumulate the list in a native view, a JavaScript view, or a
hypothetical Erlang view (i.e. a subprocess), from the operating system's
perspective, the memory for that list will be allocated somewhere. Either
the CouchDB process asks for X KB more memory, or its subprocess will ask
for it. So I think the total system impact is probably low in practice.
So I guess my point is not that native views are wrong, just they have a
cost so you should weigh the cost/benefit for your own project. In the case
of manage_couchdb, I wrote a JavaScript implementation; but since sometimes
I have an emergency and I must find conflicts ASAP, I made an Erlang
version because it is worth it.
On Thu, Aug 15, 2013 at 2:05 PM, Stanley Iriele <si...@gmail.com>wrote:
> Whoa...OK...that I had no idea about...thanks for taking the time to go to
> that granularity, by the way.
>
> So does this mean that the process memory is shared? As apposed to living
> in its own space?.so if someone accumulates a large json object in a list
> function its chewing up couchdb's memory?... I guess I'm a little confused
> about what's in the same process and what isn't now
> On Aug 14, 2013 11:57 PM, "Jason Smith" <jh...@apache.org> wrote:
>
> > To me, an Erlang view is a view server which supports map, reduce, show,
> > update, list, etc. functions in the Erlang language. (Basically it is
> > implemented in Erlang.)
> >
> > A view server is a subprocess that runs beneath CouchDB which
> communicates
> > with it over standard i/o. It is a different process in the operating
> > system and only interfaces with the main server using the view server
> > protocol (basically a bunch of JSON messages going back and forth).
> >
> > I do not know of an Erlang view server which works well and is currently
> > maintained.
> >
> > A native view (shipped by CouchDB but disabled by default) is some
> > corner-cutting. Code is evaluated directly by the primary CouchDB server.
> > Since CouchDB is Erlang, the native query server is necessarily Erlang.
> The
> > key difference is, your code is right there in the eye of the storm. You
> > can call couch_server:open("some_db") and completely circumvent security
> > and other invariants which CouchDB enforces. You can leak memory until
> the
> > kernel OOM killer terminates CouchDB. It's not about the language, it's
> > that is is running inside the CouchDB process.
> >
> >
> >
> > On Thu, Aug 15, 2013 at 1:36 PM, Stanley Iriele <siriele2x3@gmail.com
> > >wrote:
> >
> > > Wait....I'm a tad confused here..Jason what is the difference between
> > > native views and Erlang views?...
> > > On Aug 14, 2013 11:16 PM, "Jason Smith" <jh...@apache.org> wrote:
> > >
> > > > Oh, also:
> > > >
> > > > They are **not** Erlang views. They are **native** views. We should
> > > > emphasize the latter to remind ourselves about the security and
> > > reliability
> > > > risks which Bob identifies.
> > > >
> > > > They are very powerful, but it is a trade-off. Once I had a customer
> > who
> > > > had a basic "class" document describing common values. All other
> > > documents
> > > > were for modifications to the "base class" so to speak. He needed to
> > > query
> > > > by document ID, but if no such document existed, return the "base
> > class"
> > > > document instead. The product was already in the field and so the
> code
> > > > could not change. We had to change it in CouchDB.
> > > >
> > > > The fix was very simple: a _rewrite rule to a native _show function.
> In
> > > the
> > > > show function, if the Doc was null, then we used the internal CouchDB
> > API
> > > > to fetch the default document. Voila.
> > > >
> > > >
> > > > On Thu, Aug 15, 2013 at 1:08 PM, Jason Smith <jh...@apache.org> wrote:
> > > >
> > > > > On Thursday, August 15, 2013, Andrey Kuprianov wrote:
> > > > >
> > > > >> Doesnt server performance downgrade, while views are being
> rebuilt?
> > So
> > > > the
> > > > >> faster they are rebuilt, the better for you.
> > > > >
> > > > >
> > > > > If my view build would degrade total performance to cross an
> > > unacceptable
> > > > > threshold, then I am really riding the line! What about an
> unplanned
> > > > > compaction? What if one day the clients have a bug and load
> > increases?
> > > > What
> > > > > if an unplanned disaster happens and a backup must be performed
> > > urgently?
> > > > >
> > > > > I would evaluate view performance in the larger context of the
> entire
> > > > > application life cycle.
> > > > >
> > > > > Men seem to want to date beautiful women. It is a very high
> priority
> > at
> > > > > the pub or whatever. But long-married men do not even think about
> > their
> > > > > wife's attractiveness because that is a small, superficial part of
> a
> > > much
> > > > > larger story.
> > > > >
> > > > >
> > > > >>
> > > > >> Besides, looks like it's possible to do the same 3 steps with
> design
> > > doc
> > > > >> views created in Erlang? Or is it just about using require() in
> > > Node.js?
> > > > >>
> > > > >
> > > > > Actually, yes that is a fine point. I myself prefer Node.js but
> > anyone
> > > > can
> > > > > choose the best fit for them.
> > > > >
> > > > > And speaking more broadly, CouchDB is a very flexible platform so
> it
> > is
> > > > > quite likely that my own policies do not apply to every use case.
> In
> > > fact
> > > > > if I'm honest I use native views myself, usually for unplanned
> > > > > troubleshooting, I want to find conflicts so I use manage_couchdb:
> > > > > http://github.com/iriscouch/manage_couchdb
> > > > >
> > > > > My main point is, anybody time somebody says "performance" ask
> > yourself
> > > > if
> > > > > it is really a "performance siren." Earlier in this thread, Jens
> > raises
> > > > > some examples of plausible true performance requirements, not just
> > > siren
> > > > > songs.
> > > > >
> > > >
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Stanley Iriele <si...@gmail.com>.
Whoa...OK...that I had no idea about...thanks for taking the time to go to
that granularity, by the way.
So does this mean that the process memory is shared? As apposed to living
in its own space?.so if someone accumulates a large json object in a list
function its chewing up couchdb's memory?... I guess I'm a little confused
about what's in the same process and what isn't now
On Aug 14, 2013 11:57 PM, "Jason Smith" <jh...@apache.org> wrote:
> To me, an Erlang view is a view server which supports map, reduce, show,
> update, list, etc. functions in the Erlang language. (Basically it is
> implemented in Erlang.)
>
> A view server is a subprocess that runs beneath CouchDB which communicates
> with it over standard i/o. It is a different process in the operating
> system and only interfaces with the main server using the view server
> protocol (basically a bunch of JSON messages going back and forth).
>
> I do not know of an Erlang view server which works well and is currently
> maintained.
>
> A native view (shipped by CouchDB but disabled by default) is some
> corner-cutting. Code is evaluated directly by the primary CouchDB server.
> Since CouchDB is Erlang, the native query server is necessarily Erlang. The
> key difference is, your code is right there in the eye of the storm. You
> can call couch_server:open("some_db") and completely circumvent security
> and other invariants which CouchDB enforces. You can leak memory until the
> kernel OOM killer terminates CouchDB. It's not about the language, it's
> that is is running inside the CouchDB process.
>
>
>
> On Thu, Aug 15, 2013 at 1:36 PM, Stanley Iriele <siriele2x3@gmail.com
> >wrote:
>
> > Wait....I'm a tad confused here..Jason what is the difference between
> > native views and Erlang views?...
> > On Aug 14, 2013 11:16 PM, "Jason Smith" <jh...@apache.org> wrote:
> >
> > > Oh, also:
> > >
> > > They are **not** Erlang views. They are **native** views. We should
> > > emphasize the latter to remind ourselves about the security and
> > reliability
> > > risks which Bob identifies.
> > >
> > > They are very powerful, but it is a trade-off. Once I had a customer
> who
> > > had a basic "class" document describing common values. All other
> > documents
> > > were for modifications to the "base class" so to speak. He needed to
> > query
> > > by document ID, but if no such document existed, return the "base
> class"
> > > document instead. The product was already in the field and so the code
> > > could not change. We had to change it in CouchDB.
> > >
> > > The fix was very simple: a _rewrite rule to a native _show function. In
> > the
> > > show function, if the Doc was null, then we used the internal CouchDB
> API
> > > to fetch the default document. Voila.
> > >
> > >
> > > On Thu, Aug 15, 2013 at 1:08 PM, Jason Smith <jh...@apache.org> wrote:
> > >
> > > > On Thursday, August 15, 2013, Andrey Kuprianov wrote:
> > > >
> > > >> Doesnt server performance downgrade, while views are being rebuilt?
> So
> > > the
> > > >> faster they are rebuilt, the better for you.
> > > >
> > > >
> > > > If my view build would degrade total performance to cross an
> > unacceptable
> > > > threshold, then I am really riding the line! What about an unplanned
> > > > compaction? What if one day the clients have a bug and load
> increases?
> > > What
> > > > if an unplanned disaster happens and a backup must be performed
> > urgently?
> > > >
> > > > I would evaluate view performance in the larger context of the entire
> > > > application life cycle.
> > > >
> > > > Men seem to want to date beautiful women. It is a very high priority
> at
> > > > the pub or whatever. But long-married men do not even think about
> their
> > > > wife's attractiveness because that is a small, superficial part of a
> > much
> > > > larger story.
> > > >
> > > >
> > > >>
> > > >> Besides, looks like it's possible to do the same 3 steps with design
> > doc
> > > >> views created in Erlang? Or is it just about using require() in
> > Node.js?
> > > >>
> > > >
> > > > Actually, yes that is a fine point. I myself prefer Node.js but
> anyone
> > > can
> > > > choose the best fit for them.
> > > >
> > > > And speaking more broadly, CouchDB is a very flexible platform so it
> is
> > > > quite likely that my own policies do not apply to every use case. In
> > fact
> > > > if I'm honest I use native views myself, usually for unplanned
> > > > troubleshooting, I want to find conflicts so I use manage_couchdb:
> > > > http://github.com/iriscouch/manage_couchdb
> > > >
> > > > My main point is, anybody time somebody says "performance" ask
> yourself
> > > if
> > > > it is really a "performance siren." Earlier in this thread, Jens
> raises
> > > > some examples of plausible true performance requirements, not just
> > siren
> > > > songs.
> > > >
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
To me, an Erlang view is a view server which supports map, reduce, show,
update, list, etc. functions in the Erlang language. (Basically it is
implemented in Erlang.)
A view server is a subprocess that runs beneath CouchDB which communicates
with it over standard i/o. It is a different process in the operating
system and only interfaces with the main server using the view server
protocol (basically a bunch of JSON messages going back and forth).
I do not know of an Erlang view server which works well and is currently
maintained.
A native view (shipped by CouchDB but disabled by default) is some
corner-cutting. Code is evaluated directly by the primary CouchDB server.
Since CouchDB is Erlang, the native query server is necessarily Erlang. The
key difference is, your code is right there in the eye of the storm. You
can call couch_server:open("some_db") and completely circumvent security
and other invariants which CouchDB enforces. You can leak memory until the
kernel OOM killer terminates CouchDB. It's not about the language, it's
that is is running inside the CouchDB process.
On Thu, Aug 15, 2013 at 1:36 PM, Stanley Iriele <si...@gmail.com>wrote:
> Wait....I'm a tad confused here..Jason what is the difference between
> native views and Erlang views?...
> On Aug 14, 2013 11:16 PM, "Jason Smith" <jh...@apache.org> wrote:
>
> > Oh, also:
> >
> > They are **not** Erlang views. They are **native** views. We should
> > emphasize the latter to remind ourselves about the security and
> reliability
> > risks which Bob identifies.
> >
> > They are very powerful, but it is a trade-off. Once I had a customer who
> > had a basic "class" document describing common values. All other
> documents
> > were for modifications to the "base class" so to speak. He needed to
> query
> > by document ID, but if no such document existed, return the "base class"
> > document instead. The product was already in the field and so the code
> > could not change. We had to change it in CouchDB.
> >
> > The fix was very simple: a _rewrite rule to a native _show function. In
> the
> > show function, if the Doc was null, then we used the internal CouchDB API
> > to fetch the default document. Voila.
> >
> >
> > On Thu, Aug 15, 2013 at 1:08 PM, Jason Smith <jh...@apache.org> wrote:
> >
> > > On Thursday, August 15, 2013, Andrey Kuprianov wrote:
> > >
> > >> Doesnt server performance downgrade, while views are being rebuilt? So
> > the
> > >> faster they are rebuilt, the better for you.
> > >
> > >
> > > If my view build would degrade total performance to cross an
> unacceptable
> > > threshold, then I am really riding the line! What about an unplanned
> > > compaction? What if one day the clients have a bug and load increases?
> > What
> > > if an unplanned disaster happens and a backup must be performed
> urgently?
> > >
> > > I would evaluate view performance in the larger context of the entire
> > > application life cycle.
> > >
> > > Men seem to want to date beautiful women. It is a very high priority at
> > > the pub or whatever. But long-married men do not even think about their
> > > wife's attractiveness because that is a small, superficial part of a
> much
> > > larger story.
> > >
> > >
> > >>
> > >> Besides, looks like it's possible to do the same 3 steps with design
> doc
> > >> views created in Erlang? Or is it just about using require() in
> Node.js?
> > >>
> > >
> > > Actually, yes that is a fine point. I myself prefer Node.js but anyone
> > can
> > > choose the best fit for them.
> > >
> > > And speaking more broadly, CouchDB is a very flexible platform so it is
> > > quite likely that my own policies do not apply to every use case. In
> fact
> > > if I'm honest I use native views myself, usually for unplanned
> > > troubleshooting, I want to find conflicts so I use manage_couchdb:
> > > http://github.com/iriscouch/manage_couchdb
> > >
> > > My main point is, anybody time somebody says "performance" ask yourself
> > if
> > > it is really a "performance siren." Earlier in this thread, Jens raises
> > > some examples of plausible true performance requirements, not just
> siren
> > > songs.
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Stanley Iriele <si...@gmail.com>.
Wait....I'm a tad confused here..Jason what is the difference between
native views and Erlang views?...
On Aug 14, 2013 11:16 PM, "Jason Smith" <jh...@apache.org> wrote:
> Oh, also:
>
> They are **not** Erlang views. They are **native** views. We should
> emphasize the latter to remind ourselves about the security and reliability
> risks which Bob identifies.
>
> They are very powerful, but it is a trade-off. Once I had a customer who
> had a basic "class" document describing common values. All other documents
> were for modifications to the "base class" so to speak. He needed to query
> by document ID, but if no such document existed, return the "base class"
> document instead. The product was already in the field and so the code
> could not change. We had to change it in CouchDB.
>
> The fix was very simple: a _rewrite rule to a native _show function. In the
> show function, if the Doc was null, then we used the internal CouchDB API
> to fetch the default document. Voila.
>
>
> On Thu, Aug 15, 2013 at 1:08 PM, Jason Smith <jh...@apache.org> wrote:
>
> > On Thursday, August 15, 2013, Andrey Kuprianov wrote:
> >
> >> Doesnt server performance downgrade, while views are being rebuilt? So
> the
> >> faster they are rebuilt, the better for you.
> >
> >
> > If my view build would degrade total performance to cross an unacceptable
> > threshold, then I am really riding the line! What about an unplanned
> > compaction? What if one day the clients have a bug and load increases?
> What
> > if an unplanned disaster happens and a backup must be performed urgently?
> >
> > I would evaluate view performance in the larger context of the entire
> > application life cycle.
> >
> > Men seem to want to date beautiful women. It is a very high priority at
> > the pub or whatever. But long-married men do not even think about their
> > wife's attractiveness because that is a small, superficial part of a much
> > larger story.
> >
> >
> >>
> >> Besides, looks like it's possible to do the same 3 steps with design doc
> >> views created in Erlang? Or is it just about using require() in Node.js?
> >>
> >
> > Actually, yes that is a fine point. I myself prefer Node.js but anyone
> can
> > choose the best fit for them.
> >
> > And speaking more broadly, CouchDB is a very flexible platform so it is
> > quite likely that my own policies do not apply to every use case. In fact
> > if I'm honest I use native views myself, usually for unplanned
> > troubleshooting, I want to find conflicts so I use manage_couchdb:
> > http://github.com/iriscouch/manage_couchdb
> >
> > My main point is, anybody time somebody says "performance" ask yourself
> if
> > it is really a "performance siren." Earlier in this thread, Jens raises
> > some examples of plausible true performance requirements, not just siren
> > songs.
> >
>
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
Oh, also:
They are **not** Erlang views. They are **native** views. We should
emphasize the latter to remind ourselves about the security and reliability
risks which Bob identifies.
They are very powerful, but it is a trade-off. Once I had a customer who
had a basic "class" document describing common values. All other documents
were for modifications to the "base class" so to speak. He needed to query
by document ID, but if no such document existed, return the "base class"
document instead. The product was already in the field and so the code
could not change. We had to change it in CouchDB.
The fix was very simple: a _rewrite rule to a native _show function. In the
show function, if the Doc was null, then we used the internal CouchDB API
to fetch the default document. Voila.
On Thu, Aug 15, 2013 at 1:08 PM, Jason Smith <jh...@apache.org> wrote:
> On Thursday, August 15, 2013, Andrey Kuprianov wrote:
>
>> Doesnt server performance downgrade, while views are being rebuilt? So the
>> faster they are rebuilt, the better for you.
>
>
> If my view build would degrade total performance to cross an unacceptable
> threshold, then I am really riding the line! What about an unplanned
> compaction? What if one day the clients have a bug and load increases? What
> if an unplanned disaster happens and a backup must be performed urgently?
>
> I would evaluate view performance in the larger context of the entire
> application life cycle.
>
> Men seem to want to date beautiful women. It is a very high priority at
> the pub or whatever. But long-married men do not even think about their
> wife's attractiveness because that is a small, superficial part of a much
> larger story.
>
>
>>
>> Besides, looks like it's possible to do the same 3 steps with design doc
>> views created in Erlang? Or is it just about using require() in Node.js?
>>
>
> Actually, yes that is a fine point. I myself prefer Node.js but anyone can
> choose the best fit for them.
>
> And speaking more broadly, CouchDB is a very flexible platform so it is
> quite likely that my own policies do not apply to every use case. In fact
> if I'm honest I use native views myself, usually for unplanned
> troubleshooting, I want to find conflicts so I use manage_couchdb:
> http://github.com/iriscouch/manage_couchdb
>
> My main point is, anybody time somebody says "performance" ask yourself if
> it is really a "performance siren." Earlier in this thread, Jens raises
> some examples of plausible true performance requirements, not just siren
> songs.
>
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
On Thursday, August 15, 2013, Andrey Kuprianov wrote:
> Doesnt server performance downgrade, while views are being rebuilt? So the
> faster they are rebuilt, the better for you.
If my view build would degrade total performance to cross an unacceptable
threshold, then I am really riding the line! What about an unplanned
compaction? What if one day the clients have a bug and load increases? What
if an unplanned disaster happens and a backup must be performed urgently?
I would evaluate view performance in the larger context of the entire
application life cycle.
Men seem to want to date beautiful women. It is a very high priority at the
pub or whatever. But long-married men do not even think about their wife's
attractiveness because that is a small, superficial part of a much larger
story.
>
> Besides, looks like it's possible to do the same 3 steps with design doc
> views created in Erlang? Or is it just about using require() in Node.js?
>
Actually, yes that is a fine point. I myself prefer Node.js but anyone can
choose the best fit for them.
And speaking more broadly, CouchDB is a very flexible platform so it is
quite likely that my own policies do not apply to every use case. In fact
if I'm honest I use native views myself, usually for unplanned
troubleshooting, I want to find conflicts so I use manage_couchdb:
http://github.com/iriscouch/manage_couchdb
My main point is, anybody time somebody says "performance" ask yourself if
it is really a "performance siren." Earlier in this thread, Jens raises
some examples of plausible true performance requirements, not just siren
songs.
Re: Erlang vs JavaScript
Posted by Andrey Kuprianov <an...@gmail.com>.
Doesnt server performance downgrade, while views are being rebuilt? So the
faster they are rebuilt, the better for you.
Besides, looks like it's possible to do the same 3 steps with design doc
views created in Erlang? Or is it just about using require() in Node.js?
On Thu, Aug 15, 2013 at 10:52 AM, Jason Smith <jh...@apache.org> wrote:
> The time complexity of Erlang vs. JavaScript views is the same, O(n) where
> n is the document count.
>
> View building time feels important during development but it is irrelevant
> during operation. Production servers never hold up requests to build views,
> so the performance is secondary. For very important views, I prefer
> JavaScript because I can unit test them with Node.js. I don't care how long
> the view takes if I am not confident about correctness.
>
> In practice, to use views in production, I use the "PPP trick" (Push, Prep,
> Promote).
>
> ## Push
>
> 1. Use Node.js to require() my code and run some unit tests on map() and
> reduce()
> 2. Push the new design document to the server with a new, temporary ID,
> e.g. _design/foo-staging
>
> ## Prep
>
> 1. Query the view and wait for it to build
> 2. (Prep) Confirm the view output, manual or automated. I use more Node.js
> tests here
>
> ## Promote
>
> 1. curl -I <new_id> ; curl -I <old_id> # Upper-case India
> 2. Note the new etag and old etag (they are the documents' _rev values)
> 3. curl --fail -XCOPY <new_id>?rev=<new_rev>
> -HDestination:<old_id>?rev=<old_rev> \
> && curl -XDELETE <new_id>?rev=<new_rev> # Promote
>
> Note that the final step is an atomic operation. From the perspective of
> CouchDB clients, you perform an (a) instantaneous upgrade, which is (b)
> known to be bug-free, with (c) zero downtime.
>
> Obviously you can automate this procedure as needed. I do not update design
> documents much anymore, so I still use curl in a Bash script. YMMV.
>
>
> On Thu, Aug 15, 2013 at 10:25 AM, Andrey Kuprianov <
> andrey.kouprianov@gmail.com> wrote:
>
> > Yups, also true.
> >
> > In any case, we'll also be moving towards using Erlang-based design docs
> > now. Javascript is great for starters, but once you hit several dozen
> > design docs and a few hundred thousand documents, you definitely need
> > something faster.
> >
> >
> > On Thu, Aug 15, 2013 at 10:14 AM, Stanley Iriele <siriele2x3@gmail.com
> > >wrote:
> >
> > > True... But the only way for that to happen is for them to bypass
> several
> > > other layers of security... In which case I'd have much bigger problems
> > to
> > > worry about
> > > On Aug 14, 2013 8:08 PM, "Andrey Kuprianov" <
> andrey.kouprianov@gmail.com
> > >
> > > wrote:
> > >
> > > > So far it seems very promising. :) However, what they say can happen
> is
> > > > that since Erlang code has no sandbox, anything could be done to the
> > > system
> > > > through design views, if your database is compromised.
> > > >
> > > >
> > > > On Thu, Aug 15, 2013 at 9:58 AM, Stanley Iriele <
> siriele2x3@gmail.com
> > > > >wrote:
> > > >
> > > > > 5-6 times eh?...that would give me an honest reason for picking up
> > > Erlang
> > > > > again....I'm going to try converting all of my views to Erlang and
> > see
> > > > what
> > > > > happens..what's the worst that can happen? :-)
> > > > > On Aug 14, 2013 7:55 PM, "Andrey Kuprianov" <
> > > andrey.kouprianov@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > I had a 5-6 times speed boost yesterday, after converting one of
> my
> > > > > design
> > > > > > docs to Erlang. Like a breath of fresh air, if you ask me.
> > > > > >
> > > > > > Andrey
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 14, 2013 at 11:54 PM, Stanley Iriele <
> > > siriele2x3@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > I'm curious... What exactly is the performance gain from
> writing
> > > > > > > views/shows/ etc in Erlang vs JavaScript... From what I've seen
> > the
> > > > > > > JavaScript view server is sufficiently fast and feature
> complete.
> > > > > > >
> > > > > > > So my question is this.... Is the Erlang view server that much
> > > > > faster?...
> > > > > > > Is there another gain that I'm missing?..and is it as feature
> > > > complete
> > > > > as
> > > > > > > the JavaScript equivalent?
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Stanley
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Jason Smith <jh...@apache.org>.
The time complexity of Erlang vs. JavaScript views is the same, O(n) where
n is the document count.
View building time feels important during development but it is irrelevant
during operation. Production servers never hold up requests to build views,
so the performance is secondary. For very important views, I prefer
JavaScript because I can unit test them with Node.js. I don't care how long
the view takes if I am not confident about correctness.
In practice, to use views in production, I use the "PPP trick" (Push, Prep,
Promote).
## Push
1. Use Node.js to require() my code and run some unit tests on map() and
reduce()
2. Push the new design document to the server with a new, temporary ID,
e.g. _design/foo-staging
## Prep
1. Query the view and wait for it to build
2. (Prep) Confirm the view output, manual or automated. I use more Node.js
tests here
## Promote
1. curl -I <new_id> ; curl -I <old_id> # Upper-case India
2. Note the new etag and old etag (they are the documents' _rev values)
3. curl --fail -XCOPY <new_id>?rev=<new_rev>
-HDestination:<old_id>?rev=<old_rev> \
&& curl -XDELETE <new_id>?rev=<new_rev> # Promote
Note that the final step is an atomic operation. From the perspective of
CouchDB clients, you perform an (a) instantaneous upgrade, which is (b)
known to be bug-free, with (c) zero downtime.
Obviously you can automate this procedure as needed. I do not update design
documents much anymore, so I still use curl in a Bash script. YMMV.
On Thu, Aug 15, 2013 at 10:25 AM, Andrey Kuprianov <
andrey.kouprianov@gmail.com> wrote:
> Yups, also true.
>
> In any case, we'll also be moving towards using Erlang-based design docs
> now. Javascript is great for starters, but once you hit several dozen
> design docs and a few hundred thousand documents, you definitely need
> something faster.
>
>
> On Thu, Aug 15, 2013 at 10:14 AM, Stanley Iriele <siriele2x3@gmail.com
> >wrote:
>
> > True... But the only way for that to happen is for them to bypass several
> > other layers of security... In which case I'd have much bigger problems
> to
> > worry about
> > On Aug 14, 2013 8:08 PM, "Andrey Kuprianov" <andrey.kouprianov@gmail.com
> >
> > wrote:
> >
> > > So far it seems very promising. :) However, what they say can happen is
> > > that since Erlang code has no sandbox, anything could be done to the
> > system
> > > through design views, if your database is compromised.
> > >
> > >
> > > On Thu, Aug 15, 2013 at 9:58 AM, Stanley Iriele <siriele2x3@gmail.com
> > > >wrote:
> > >
> > > > 5-6 times eh?...that would give me an honest reason for picking up
> > Erlang
> > > > again....I'm going to try converting all of my views to Erlang and
> see
> > > what
> > > > happens..what's the worst that can happen? :-)
> > > > On Aug 14, 2013 7:55 PM, "Andrey Kuprianov" <
> > andrey.kouprianov@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I had a 5-6 times speed boost yesterday, after converting one of my
> > > > design
> > > > > docs to Erlang. Like a breath of fresh air, if you ask me.
> > > > >
> > > > > Andrey
> > > > >
> > > > >
> > > > > On Wed, Aug 14, 2013 at 11:54 PM, Stanley Iriele <
> > siriele2x3@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > I'm curious... What exactly is the performance gain from writing
> > > > > > views/shows/ etc in Erlang vs JavaScript... From what I've seen
> the
> > > > > > JavaScript view server is sufficiently fast and feature complete.
> > > > > >
> > > > > > So my question is this.... Is the Erlang view server that much
> > > > faster?...
> > > > > > Is there another gain that I'm missing?..and is it as feature
> > > complete
> > > > as
> > > > > > the JavaScript equivalent?
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Stanley
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Andrey Kuprianov <an...@gmail.com>.
Yups, also true.
In any case, we'll also be moving towards using Erlang-based design docs
now. Javascript is great for starters, but once you hit several dozen
design docs and a few hundred thousand documents, you definitely need
something faster.
On Thu, Aug 15, 2013 at 10:14 AM, Stanley Iriele <si...@gmail.com>wrote:
> True... But the only way for that to happen is for them to bypass several
> other layers of security... In which case I'd have much bigger problems to
> worry about
> On Aug 14, 2013 8:08 PM, "Andrey Kuprianov" <an...@gmail.com>
> wrote:
>
> > So far it seems very promising. :) However, what they say can happen is
> > that since Erlang code has no sandbox, anything could be done to the
> system
> > through design views, if your database is compromised.
> >
> >
> > On Thu, Aug 15, 2013 at 9:58 AM, Stanley Iriele <siriele2x3@gmail.com
> > >wrote:
> >
> > > 5-6 times eh?...that would give me an honest reason for picking up
> Erlang
> > > again....I'm going to try converting all of my views to Erlang and see
> > what
> > > happens..what's the worst that can happen? :-)
> > > On Aug 14, 2013 7:55 PM, "Andrey Kuprianov" <
> andrey.kouprianov@gmail.com
> > >
> > > wrote:
> > >
> > > > I had a 5-6 times speed boost yesterday, after converting one of my
> > > design
> > > > docs to Erlang. Like a breath of fresh air, if you ask me.
> > > >
> > > > Andrey
> > > >
> > > >
> > > > On Wed, Aug 14, 2013 at 11:54 PM, Stanley Iriele <
> siriele2x3@gmail.com
> > > > >wrote:
> > > >
> > > > > I'm curious... What exactly is the performance gain from writing
> > > > > views/shows/ etc in Erlang vs JavaScript... From what I've seen the
> > > > > JavaScript view server is sufficiently fast and feature complete.
> > > > >
> > > > > So my question is this.... Is the Erlang view server that much
> > > faster?...
> > > > > Is there another gain that I'm missing?..and is it as feature
> > complete
> > > as
> > > > > the JavaScript equivalent?
> > > > >
> > > > > Regards,
> > > > >
> > > > > Stanley
> > > > >
> > > >
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Stanley Iriele <si...@gmail.com>.
True... But the only way for that to happen is for them to bypass several
other layers of security... In which case I'd have much bigger problems to
worry about
On Aug 14, 2013 8:08 PM, "Andrey Kuprianov" <an...@gmail.com>
wrote:
> So far it seems very promising. :) However, what they say can happen is
> that since Erlang code has no sandbox, anything could be done to the system
> through design views, if your database is compromised.
>
>
> On Thu, Aug 15, 2013 at 9:58 AM, Stanley Iriele <siriele2x3@gmail.com
> >wrote:
>
> > 5-6 times eh?...that would give me an honest reason for picking up Erlang
> > again....I'm going to try converting all of my views to Erlang and see
> what
> > happens..what's the worst that can happen? :-)
> > On Aug 14, 2013 7:55 PM, "Andrey Kuprianov" <andrey.kouprianov@gmail.com
> >
> > wrote:
> >
> > > I had a 5-6 times speed boost yesterday, after converting one of my
> > design
> > > docs to Erlang. Like a breath of fresh air, if you ask me.
> > >
> > > Andrey
> > >
> > >
> > > On Wed, Aug 14, 2013 at 11:54 PM, Stanley Iriele <siriele2x3@gmail.com
> > > >wrote:
> > >
> > > > I'm curious... What exactly is the performance gain from writing
> > > > views/shows/ etc in Erlang vs JavaScript... From what I've seen the
> > > > JavaScript view server is sufficiently fast and feature complete.
> > > >
> > > > So my question is this.... Is the Erlang view server that much
> > faster?...
> > > > Is there another gain that I'm missing?..and is it as feature
> complete
> > as
> > > > the JavaScript equivalent?
> > > >
> > > > Regards,
> > > >
> > > > Stanley
> > > >
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Andrey Kuprianov <an...@gmail.com>.
So far it seems very promising. :) However, what they say can happen is
that since Erlang code has no sandbox, anything could be done to the system
through design views, if your database is compromised.
On Thu, Aug 15, 2013 at 9:58 AM, Stanley Iriele <si...@gmail.com>wrote:
> 5-6 times eh?...that would give me an honest reason for picking up Erlang
> again....I'm going to try converting all of my views to Erlang and see what
> happens..what's the worst that can happen? :-)
> On Aug 14, 2013 7:55 PM, "Andrey Kuprianov" <an...@gmail.com>
> wrote:
>
> > I had a 5-6 times speed boost yesterday, after converting one of my
> design
> > docs to Erlang. Like a breath of fresh air, if you ask me.
> >
> > Andrey
> >
> >
> > On Wed, Aug 14, 2013 at 11:54 PM, Stanley Iriele <siriele2x3@gmail.com
> > >wrote:
> >
> > > I'm curious... What exactly is the performance gain from writing
> > > views/shows/ etc in Erlang vs JavaScript... From what I've seen the
> > > JavaScript view server is sufficiently fast and feature complete.
> > >
> > > So my question is this.... Is the Erlang view server that much
> faster?...
> > > Is there another gain that I'm missing?..and is it as feature complete
> as
> > > the JavaScript equivalent?
> > >
> > > Regards,
> > >
> > > Stanley
> > >
> >
>
Re: Erlang vs JavaScript
Posted by Stanley Iriele <si...@gmail.com>.
5-6 times eh?...that would give me an honest reason for picking up Erlang
again....I'm going to try converting all of my views to Erlang and see what
happens..what's the worst that can happen? :-)
On Aug 14, 2013 7:55 PM, "Andrey Kuprianov" <an...@gmail.com>
wrote:
> I had a 5-6 times speed boost yesterday, after converting one of my design
> docs to Erlang. Like a breath of fresh air, if you ask me.
>
> Andrey
>
>
> On Wed, Aug 14, 2013 at 11:54 PM, Stanley Iriele <siriele2x3@gmail.com
> >wrote:
>
> > I'm curious... What exactly is the performance gain from writing
> > views/shows/ etc in Erlang vs JavaScript... From what I've seen the
> > JavaScript view server is sufficiently fast and feature complete.
> >
> > So my question is this.... Is the Erlang view server that much faster?...
> > Is there another gain that I'm missing?..and is it as feature complete as
> > the JavaScript equivalent?
> >
> > Regards,
> >
> > Stanley
> >
>
Re: Erlang vs JavaScript
Posted by Andrey Kuprianov <an...@gmail.com>.
I had a 5-6 times speed boost yesterday, after converting one of my design
docs to Erlang. Like a breath of fresh air, if you ask me.
Andrey
On Wed, Aug 14, 2013 at 11:54 PM, Stanley Iriele <si...@gmail.com>wrote:
> I'm curious... What exactly is the performance gain from writing
> views/shows/ etc in Erlang vs JavaScript... From what I've seen the
> JavaScript view server is sufficiently fast and feature complete.
>
> So my question is this.... Is the Erlang view server that much faster?...
> Is there another gain that I'm missing?..and is it as feature complete as
> the JavaScript equivalent?
>
> Regards,
>
> Stanley
>
Re: Erlang vs JavaScript
Posted by Jens Alfke <je...@couchbase.com>.
On Aug 14, 2013, at 1:57 PM, Filippo Fadda <fi...@programmazione.it> wrote:
> The bottleneck is not the language used to write the Query Server, but it's the standard input/output used for inter-process communication.
That’s basically what I’m saying, if you include the JSON marshaling in the stdio overhead.
> Why not use 0MQ instead?
Why would that be faster? Presumably it’s still marshaling the data in some form and sending it to another process over pipes or sockets. (But if it’s using something fancy like shared-memory buffers to do the IPC, that would indeed be better.)
If you use an in-process JSVM you can directly traverse the Erlang term and construct a corresponding JS object in memory without going through any intermediate representation like JSON; that’s a lot faster. (Our old iErl repo from Couchbase Mobile has code for doing this with SpiderMonkey.)
—Jens
Re: Erlang vs JavaScript
Posted by Filippo Fadda <fi...@programmazione.it>.
The bottleneck is not the language used to write the Query Server, but it's the standard input/output used for inter-process communication.
Why not use 0MQ instead?
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
On Wed, Aug 14, 2013 at 11:50 PM, Jens Alfke <je...@couchbase.com> wrote:
> On Aug 14, 2013, at 10:02 AM, Alexander Shorin <kx...@gmail.com> wrote:
>
>> Erlang server bypassed stdio interface communication and addtional
>> JSON decode/encode roundtrip, so it is faster than JS at some point.
>
> Yeah — the ironic thing is that state-of-the-art JS runtimes are probably faster than Erlang* these days, but JS views are still going to be slower because (a) they run in a separate process, and (b) the docs have to be translated from Erlang terms into JS objects. My experience from working inter-language bridges is that parameter marshaling is generally a performance killer.
I believe that views can be processed in parallel to speed up
indexing, but this will be trade-off between speed and overall server
performance (easy to hit OOM state with big docs). Also, this will
requires communication protocol changing and Samuel's already made a
proposal about:
https://issues.apache.org/jira/browse/COUCHDB-1743
https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
But I agree with your last sentence. However, it's also tradeoff
between flexibility and performance (;
> * Not intending to start a language-performance flame war! But Erlang’s interpreter is quite primitive by today’s standards: it doesn’t even have a JIT.
Offtopic joke: even PHP has JIT and dies after user request processing
for years before Erlang (:
--
,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Jens Alfke <je...@couchbase.com>.
On Aug 14, 2013, at 10:02 AM, Alexander Shorin <kx...@gmail.com> wrote:
> Erlang server bypassed stdio interface communication and addtional
> JSON decode/encode roundtrip, so it is faster than JS at some point.
Yeah — the ironic thing is that state-of-the-art JS runtimes are probably faster than Erlang* these days, but JS views are still going to be slower because (a) they run in a separate process, and (b) the docs have to be translated from Erlang terms into JS objects. My experience from working inter-language bridges is that parameter marshaling is generally a performance killer.
FWIW, Couchbase Server 2.0 uses V8 instead of SpiderMonkey and runs it in-process; both those changes really help performance.
—Jens
* Not intending to start a language-performance flame war! But Erlang’s interpreter is quite primitive by today’s standards: it doesn’t even have a JIT.
Re: Erlang vs JavaScript
Posted by Mike Marino <mm...@gmail.com>.
I've seen some reasonably good performance gains writing the views in
erlang instead of javascript (5-10 times faster), I suspect this must
depend strongly on what's being done though in the view calculation.
One thing that might be important to consider is the amount of time it
takes to develop an erlang view, which I found could be challenging to
debug (at least for me, an erlang newbie) using couch logs.
If you're interested in real world translation examples (I had trouble
finding some when I was building them), I have some javascript/erlang
comparison views here:
https://github.com/mgmarino/nEDM-Interface/blob/master/_defaulterlang/_design_erlang.json
https://github.com/mgmarino/nEDM-Interface/blob/master/_default/lib/views.js
Cheers,
Mike
On Wed, Aug 14, 2013 at 7:25 PM, Filippo Fadda <
filippo.fadda@programmazione.it> wrote:
> You can also write map and reduce functions in PHP using ElephantOnCouch
> Query Server:
>
> https://github.com/dedalozzo/EOCSvr
>
> I still don't have integrated lists and shows, but I will do it soon.
>
> There are query server implementations in other languages also.
>
> -Filippo
>
>
> On Aug 14, 2013, at 7:17 PM, Stanley Iriele wrote:
>
> > Oh wow...that is pretty snazzy....this is a bit of a dumb question but
> can
> > everything in a design doc be written in Erlang?..like list
> > functions..shows..etc?
> > On Aug 14, 2013 10:05 AM, "Robert Newson" <rn...@apache.org> wrote:
> >
> >> Erlang is faster and more feature rich. One thing you can do in an
> >> erlang view that you can't do in javascript is
> >> file:delete(AnyPathYourServerCanAccess), which is why we disable it by
> >> default. :)
> >>
> >> Building a view is inherently serial regardless of the language
> >> evaluating each update.
> >>
> >> B.
> >>
> >>
> >> On 14 August 2013 18:02, Alexander Shorin <kx...@gmail.com> wrote:
> >>> Erlang server bypassed stdio interface communication and addtional
> >>> JSON decode/encode roundtrip, so it is faster than JS at some point.
> >>> Also, having Erlang with SMP feature enabled may also improve his
> >>> performance if you have multicore server.
> >>>
> >>> How much? No numbers I have, just an experience background. You can
> >>> measure it with you functions and data and share with us (:
> >>>
> >>> --
> >>> ,,,^..^,,,
> >>>
> >>>
> >>> On Wed, Aug 14, 2013 at 8:54 PM, Stanley Iriele <si...@gmail.com>
> >> wrote:
> >>>> I'm curious... What exactly is the performance gain from writing
> >>>> views/shows/ etc in Erlang vs JavaScript... From what I've seen the
> >>>> JavaScript view server is sufficiently fast and feature complete.
> >>>>
> >>>> So my question is this.... Is the Erlang view server that much
> >> faster?...
> >>>> Is there another gain that I'm missing?..and is it as feature complete
> >> as
> >>>> the JavaScript equivalent?
> >>>>
> >>>> Regards,
> >>>>
> >>>> Stanley
> >>
>
>
Re: Erlang vs JavaScript
Posted by Filippo Fadda <fi...@programmazione.it>.
You can also write map and reduce functions in PHP using ElephantOnCouch Query Server:
https://github.com/dedalozzo/EOCSvr
I still don't have integrated lists and shows, but I will do it soon.
There are query server implementations in other languages also.
-Filippo
On Aug 14, 2013, at 7:17 PM, Stanley Iriele wrote:
> Oh wow...that is pretty snazzy....this is a bit of a dumb question but can
> everything in a design doc be written in Erlang?..like list
> functions..shows..etc?
> On Aug 14, 2013 10:05 AM, "Robert Newson" <rn...@apache.org> wrote:
>
>> Erlang is faster and more feature rich. One thing you can do in an
>> erlang view that you can't do in javascript is
>> file:delete(AnyPathYourServerCanAccess), which is why we disable it by
>> default. :)
>>
>> Building a view is inherently serial regardless of the language
>> evaluating each update.
>>
>> B.
>>
>>
>> On 14 August 2013 18:02, Alexander Shorin <kx...@gmail.com> wrote:
>>> Erlang server bypassed stdio interface communication and addtional
>>> JSON decode/encode roundtrip, so it is faster than JS at some point.
>>> Also, having Erlang with SMP feature enabled may also improve his
>>> performance if you have multicore server.
>>>
>>> How much? No numbers I have, just an experience background. You can
>>> measure it with you functions and data and share with us (:
>>>
>>> --
>>> ,,,^..^,,,
>>>
>>>
>>> On Wed, Aug 14, 2013 at 8:54 PM, Stanley Iriele <si...@gmail.com>
>> wrote:
>>>> I'm curious... What exactly is the performance gain from writing
>>>> views/shows/ etc in Erlang vs JavaScript... From what I've seen the
>>>> JavaScript view server is sufficiently fast and feature complete.
>>>>
>>>> So my question is this.... Is the Erlang view server that much
>> faster?...
>>>> Is there another gain that I'm missing?..and is it as feature complete
>> as
>>>> the JavaScript equivalent?
>>>>
>>>> Regards,
>>>>
>>>> Stanley
>>
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
On Wed, Aug 14, 2013 at 9:17 PM, Stanley Iriele <si...@gmail.com> wrote:
> Oh wow...that is pretty snazzy....this is a bit of a dumb question but can
> everything in a design doc be written in Erlang?..like list
> functions..shows..etc?
Yes, you can write any design functions on Erlang.
http://docs.couchdb.org/en/latest/query-servers.html#erlang
--
,,,^..^,,,
Re: Erlang vs JavaScript
Posted by Stanley Iriele <si...@gmail.com>.
Oh wow...that is pretty snazzy....this is a bit of a dumb question but can
everything in a design doc be written in Erlang?..like list
functions..shows..etc?
On Aug 14, 2013 10:05 AM, "Robert Newson" <rn...@apache.org> wrote:
> Erlang is faster and more feature rich. One thing you can do in an
> erlang view that you can't do in javascript is
> file:delete(AnyPathYourServerCanAccess), which is why we disable it by
> default. :)
>
> Building a view is inherently serial regardless of the language
> evaluating each update.
>
> B.
>
>
> On 14 August 2013 18:02, Alexander Shorin <kx...@gmail.com> wrote:
> > Erlang server bypassed stdio interface communication and addtional
> > JSON decode/encode roundtrip, so it is faster than JS at some point.
> > Also, having Erlang with SMP feature enabled may also improve his
> > performance if you have multicore server.
> >
> > How much? No numbers I have, just an experience background. You can
> > measure it with you functions and data and share with us (:
> >
> > --
> > ,,,^..^,,,
> >
> >
> > On Wed, Aug 14, 2013 at 8:54 PM, Stanley Iriele <si...@gmail.com>
> wrote:
> >> I'm curious... What exactly is the performance gain from writing
> >> views/shows/ etc in Erlang vs JavaScript... From what I've seen the
> >> JavaScript view server is sufficiently fast and feature complete.
> >>
> >> So my question is this.... Is the Erlang view server that much
> faster?...
> >> Is there another gain that I'm missing?..and is it as feature complete
> as
> >> the JavaScript equivalent?
> >>
> >> Regards,
> >>
> >> Stanley
>
Re: Erlang vs JavaScript
Posted by Robert Newson <rn...@apache.org>.
Erlang is faster and more feature rich. One thing you can do in an
erlang view that you can't do in javascript is
file:delete(AnyPathYourServerCanAccess), which is why we disable it by
default. :)
Building a view is inherently serial regardless of the language
evaluating each update.
B.
On 14 August 2013 18:02, Alexander Shorin <kx...@gmail.com> wrote:
> Erlang server bypassed stdio interface communication and addtional
> JSON decode/encode roundtrip, so it is faster than JS at some point.
> Also, having Erlang with SMP feature enabled may also improve his
> performance if you have multicore server.
>
> How much? No numbers I have, just an experience background. You can
> measure it with you functions and data and share with us (:
>
> --
> ,,,^..^,,,
>
>
> On Wed, Aug 14, 2013 at 8:54 PM, Stanley Iriele <si...@gmail.com> wrote:
>> I'm curious... What exactly is the performance gain from writing
>> views/shows/ etc in Erlang vs JavaScript... From what I've seen the
>> JavaScript view server is sufficiently fast and feature complete.
>>
>> So my question is this.... Is the Erlang view server that much faster?...
>> Is there another gain that I'm missing?..and is it as feature complete as
>> the JavaScript equivalent?
>>
>> Regards,
>>
>> Stanley
Re: Erlang vs JavaScript
Posted by Alexander Shorin <kx...@gmail.com>.
Erlang server bypassed stdio interface communication and addtional
JSON decode/encode roundtrip, so it is faster than JS at some point.
Also, having Erlang with SMP feature enabled may also improve his
performance if you have multicore server.
How much? No numbers I have, just an experience background. You can
measure it with you functions and data and share with us (:
--
,,,^..^,,,
On Wed, Aug 14, 2013 at 8:54 PM, Stanley Iriele <si...@gmail.com> wrote:
> I'm curious... What exactly is the performance gain from writing
> views/shows/ etc in Erlang vs JavaScript... From what I've seen the
> JavaScript view server is sufficiently fast and feature complete.
>
> So my question is this.... Is the Erlang view server that much faster?...
> Is there another gain that I'm missing?..and is it as feature complete as
> the JavaScript equivalent?
>
> Regards,
>
> Stanley