You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Caolan McMahon <ca...@gmail.com> on 2011/02/22 17:47:36 UTC
CommonJS module updates
As some of you may know, I've been working on a couchapp framework
which makes heavy use of commonjs modules (http://kansojs.org). While
developing this I've run into a number of issues which prevent the use
of some modules, and makes writing my own more difficult:
1. Modules are not cached - eval'ing a complex application, consisting
of many modules on each request would have an impact on performance.
It also means you can't use modules which use the module object to
store state. This is commonly used by template libraries to store
loaded templates in a cache, or 'memoize' expensive functions.
2. Circular dependencies blow the stack - Its not possible to require
module A from module B, if module B also requires module A. This
happens more often than you might think, and is handled by other
require() implementations by setting the cached module to an empty
object before eval'ing it. The fix for this requires a module cache to
be in place.
Since these are really hindering progress, I've forked on github and
committed my proposed patches with associated tests:
https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
Please, if you have time, review the code and provide me with your feedback.
Thanks,
Caolan
Re: CommonJS module updates
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Feb 22, 2011 at 12:20 PM, Caolan McMahon
<ca...@gmail.com> wrote:
> I can see the case for not wanting to store state on modules between
> requests, as this could break caching rules for those resources. Even
> though there is a performance hit. How would you know if the results
> for a show or list function had changed?
>
> It also sounds like this would be unreliable anyway since there are
> multiple JS processes. However, I think we must have caching between
> require()'s within a single request, as otherwise some modules will
> not work and we can't handle circular dependencies.
>
> The question is, should CouchDB enforce the fact that applications
> should not store state between requests (except for caching) and take
> the performance hit, or should this be left to commonjs module
> developers?
>
Yeah, I'm not entirely sure on this point. Technically you can already
do some pretty weird things if you're determined enough, but with the
module cacheing I think that line becomes less obvious.
> I assume freezing modules would cancel out the benefits of increased
> module compatability...
>
Yeah, I don't think its the perfect solution either. It seems like the
sort of thing that would randomly break various modules.
>
> On 22 February 2011 16:57, Paul Davis <pa...@gmail.com> wrote:
>> On Tue, Feb 22, 2011 at 11:47 AM, Caolan McMahon
>> <ca...@gmail.com> wrote:
>>> As some of you may know, I've been working on a couchapp framework
>>> which makes heavy use of commonjs modules (http://kansojs.org). While
>>> developing this I've run into a number of issues which prevent the use
>>> of some modules, and makes writing my own more difficult:
>>>
>>> 1. Modules are not cached - eval'ing a complex application, consisting
>>> of many modules on each request would have an impact on performance.
>>> It also means you can't use modules which use the module object to
>>> store state. This is commonly used by template libraries to store
>>> loaded templates in a cache, or 'memoize' expensive functions.
>>>
>>> 2. Circular dependencies blow the stack - Its not possible to require
>>> module A from module B, if module B also requires module A. This
>>> happens more often than you might think, and is handled by other
>>> require() implementations by setting the cached module to an empty
>>> object before eval'ing it. The fix for this requires a module cache to
>>> be in place.
>>>
>>>
>>> Since these are really hindering progress, I've forked on github and
>>> committed my proposed patches with associated tests:
>>> https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
>>>
>>> Please, if you have time, review the code and provide me with your feedback.
>>>
>>>
>>> Thanks,
>>>
>>> Caolan
>>>
>>
>> The circular dependency section looks good.
>>
>> The bit on caching and testing that things are cached is not good on
>> the other hand. The way that JS processes are used you can never be
>> sure if it'll be the same os process handling the request. In the test
>> suite, its more than likely to be the same OS process because of how
>> the server gets restarted often and there's a single serialized
>> client.
>>
>> I'm a bit iffy on whether we should cache modules because that'd be a
>> pretty easy place to break view updates and could lead to other weird
>> bits in the show/list stuff. Though I also understand the concern for
>> avoiding all the recompilation. I wonder, is it possible to freeze the
>> module maybe?
>>
>
Re: CommonJS module updates
Posted by Caolan McMahon <ca...@gmail.com>.
I've added a ticket in JIRA for the circular require issue:
https://issues.apache.org/jira/browse/COUCHDB-1075
I've also commented on the module cache ticket Filipe mentioned:
https://issues.apache.org/jira/browse/COUCHDB-890
If anyone has time to review the patch attached to the first ticket
I'd really appreciate it!
Thanks,
Caolan
On 22 February 2011 19:12, Caolan McMahon <ca...@gmail.com> wrote:
> Thanks for the pointer ;)
>
> https://github.com/caolan/couchdb/commit/439b6a65a4326b90307c5e0e81beacb249461420
>
> I'm starting to think this is the safest option until we've had chance
> to more widely discuss the options regarding a more persistent module
> cache.
>
>
>
> On 22 February 2011 18:14, Paul Davis <pa...@gmail.com> wrote:
>> On Tue, Feb 22, 2011 at 1:07 PM, Caolan McMahon
>> <ca...@gmail.com> wrote:
>>> An alternative, which avoids caching modules between requests:
>>> https://github.com/caolan/couchdb/commit/76c4f8ef9ab719b3d5cae22255197d06d1395756
>>>
>>> I'd prefer if we did cache modules between requests, but this would
>>> still solve my problems ;)
>>>
>>>
>>> On 22 February 2011 17:20, Caolan McMahon <ca...@gmail.com> wrote:
>>>> I can see the case for not wanting to store state on modules between
>>>> requests, as this could break caching rules for those resources. Even
>>>> though there is a performance hit. How would you know if the results
>>>> for a show or list function had changed?
>>>>
>>>> It also sounds like this would be unreliable anyway since there are
>>>> multiple JS processes. However, I think we must have caching between
>>>> require()'s within a single request, as otherwise some modules will
>>>> not work and we can't handle circular dependencies.
>>>>
>>>> The question is, should CouchDB enforce the fact that applications
>>>> should not store state between requests (except for caching) and take
>>>> the performance hit, or should this be left to commonjs module
>>>> developers?
>>>>
>>>> I assume freezing modules would cancel out the benefits of increased
>>>> module compatability...
>>>>
>>>>
>>>> On 22 February 2011 16:57, Paul Davis <pa...@gmail.com> wrote:
>>>>> On Tue, Feb 22, 2011 at 11:47 AM, Caolan McMahon
>>>>> <ca...@gmail.com> wrote:
>>>>>> As some of you may know, I've been working on a couchapp framework
>>>>>> which makes heavy use of commonjs modules (http://kansojs.org). While
>>>>>> developing this I've run into a number of issues which prevent the use
>>>>>> of some modules, and makes writing my own more difficult:
>>>>>>
>>>>>> 1. Modules are not cached - eval'ing a complex application, consisting
>>>>>> of many modules on each request would have an impact on performance.
>>>>>> It also means you can't use modules which use the module object to
>>>>>> store state. This is commonly used by template libraries to store
>>>>>> loaded templates in a cache, or 'memoize' expensive functions.
>>>>>>
>>>>>> 2. Circular dependencies blow the stack - Its not possible to require
>>>>>> module A from module B, if module B also requires module A. This
>>>>>> happens more often than you might think, and is handled by other
>>>>>> require() implementations by setting the cached module to an empty
>>>>>> object before eval'ing it. The fix for this requires a module cache to
>>>>>> be in place.
>>>>>>
>>>>>>
>>>>>> Since these are really hindering progress, I've forked on github and
>>>>>> committed my proposed patches with associated tests:
>>>>>> https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
>>>>>>
>>>>>> Please, if you have time, review the code and provide me with your feedback.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Caolan
>>>>>>
>>>>>
>>>>> The circular dependency section looks good.
>>>>>
>>>>> The bit on caching and testing that things are cached is not good on
>>>>> the other hand. The way that JS processes are used you can never be
>>>>> sure if it'll be the same os process handling the request. In the test
>>>>> suite, its more than likely to be the same OS process because of how
>>>>> the server gets restarted often and there's a single serialized
>>>>> client.
>>>>>
>>>>> I'm a bit iffy on whether we should cache modules because that'd be a
>>>>> pretty easy place to break view updates and could lead to other weird
>>>>> bits in the show/list stuff. Though I also understand the concern for
>>>>> avoiding all the recompilation. I wonder, is it possible to freeze the
>>>>> module maybe?
>>>>>
>>>>
>>>
>>
>> Looks good on first pass, though the resetModuleCache() might be
>> better off being put into the handler for the reset command.
>>
>
Re: CommonJS module updates
Posted by Caolan McMahon <ca...@gmail.com>.
Thanks for the pointer ;)
https://github.com/caolan/couchdb/commit/439b6a65a4326b90307c5e0e81beacb249461420
I'm starting to think this is the safest option until we've had chance
to more widely discuss the options regarding a more persistent module
cache.
On 22 February 2011 18:14, Paul Davis <pa...@gmail.com> wrote:
> On Tue, Feb 22, 2011 at 1:07 PM, Caolan McMahon
> <ca...@gmail.com> wrote:
>> An alternative, which avoids caching modules between requests:
>> https://github.com/caolan/couchdb/commit/76c4f8ef9ab719b3d5cae22255197d06d1395756
>>
>> I'd prefer if we did cache modules between requests, but this would
>> still solve my problems ;)
>>
>>
>> On 22 February 2011 17:20, Caolan McMahon <ca...@gmail.com> wrote:
>>> I can see the case for not wanting to store state on modules between
>>> requests, as this could break caching rules for those resources. Even
>>> though there is a performance hit. How would you know if the results
>>> for a show or list function had changed?
>>>
>>> It also sounds like this would be unreliable anyway since there are
>>> multiple JS processes. However, I think we must have caching between
>>> require()'s within a single request, as otherwise some modules will
>>> not work and we can't handle circular dependencies.
>>>
>>> The question is, should CouchDB enforce the fact that applications
>>> should not store state between requests (except for caching) and take
>>> the performance hit, or should this be left to commonjs module
>>> developers?
>>>
>>> I assume freezing modules would cancel out the benefits of increased
>>> module compatability...
>>>
>>>
>>> On 22 February 2011 16:57, Paul Davis <pa...@gmail.com> wrote:
>>>> On Tue, Feb 22, 2011 at 11:47 AM, Caolan McMahon
>>>> <ca...@gmail.com> wrote:
>>>>> As some of you may know, I've been working on a couchapp framework
>>>>> which makes heavy use of commonjs modules (http://kansojs.org). While
>>>>> developing this I've run into a number of issues which prevent the use
>>>>> of some modules, and makes writing my own more difficult:
>>>>>
>>>>> 1. Modules are not cached - eval'ing a complex application, consisting
>>>>> of many modules on each request would have an impact on performance.
>>>>> It also means you can't use modules which use the module object to
>>>>> store state. This is commonly used by template libraries to store
>>>>> loaded templates in a cache, or 'memoize' expensive functions.
>>>>>
>>>>> 2. Circular dependencies blow the stack - Its not possible to require
>>>>> module A from module B, if module B also requires module A. This
>>>>> happens more often than you might think, and is handled by other
>>>>> require() implementations by setting the cached module to an empty
>>>>> object before eval'ing it. The fix for this requires a module cache to
>>>>> be in place.
>>>>>
>>>>>
>>>>> Since these are really hindering progress, I've forked on github and
>>>>> committed my proposed patches with associated tests:
>>>>> https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
>>>>>
>>>>> Please, if you have time, review the code and provide me with your feedback.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Caolan
>>>>>
>>>>
>>>> The circular dependency section looks good.
>>>>
>>>> The bit on caching and testing that things are cached is not good on
>>>> the other hand. The way that JS processes are used you can never be
>>>> sure if it'll be the same os process handling the request. In the test
>>>> suite, its more than likely to be the same OS process because of how
>>>> the server gets restarted often and there's a single serialized
>>>> client.
>>>>
>>>> I'm a bit iffy on whether we should cache modules because that'd be a
>>>> pretty easy place to break view updates and could lead to other weird
>>>> bits in the show/list stuff. Though I also understand the concern for
>>>> avoiding all the recompilation. I wonder, is it possible to freeze the
>>>> module maybe?
>>>>
>>>
>>
>
> Looks good on first pass, though the resetModuleCache() might be
> better off being put into the handler for the reset command.
>
Re: CommonJS module updates
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Feb 22, 2011 at 1:07 PM, Caolan McMahon
<ca...@gmail.com> wrote:
> An alternative, which avoids caching modules between requests:
> https://github.com/caolan/couchdb/commit/76c4f8ef9ab719b3d5cae22255197d06d1395756
>
> I'd prefer if we did cache modules between requests, but this would
> still solve my problems ;)
>
>
> On 22 February 2011 17:20, Caolan McMahon <ca...@gmail.com> wrote:
>> I can see the case for not wanting to store state on modules between
>> requests, as this could break caching rules for those resources. Even
>> though there is a performance hit. How would you know if the results
>> for a show or list function had changed?
>>
>> It also sounds like this would be unreliable anyway since there are
>> multiple JS processes. However, I think we must have caching between
>> require()'s within a single request, as otherwise some modules will
>> not work and we can't handle circular dependencies.
>>
>> The question is, should CouchDB enforce the fact that applications
>> should not store state between requests (except for caching) and take
>> the performance hit, or should this be left to commonjs module
>> developers?
>>
>> I assume freezing modules would cancel out the benefits of increased
>> module compatability...
>>
>>
>> On 22 February 2011 16:57, Paul Davis <pa...@gmail.com> wrote:
>>> On Tue, Feb 22, 2011 at 11:47 AM, Caolan McMahon
>>> <ca...@gmail.com> wrote:
>>>> As some of you may know, I've been working on a couchapp framework
>>>> which makes heavy use of commonjs modules (http://kansojs.org). While
>>>> developing this I've run into a number of issues which prevent the use
>>>> of some modules, and makes writing my own more difficult:
>>>>
>>>> 1. Modules are not cached - eval'ing a complex application, consisting
>>>> of many modules on each request would have an impact on performance.
>>>> It also means you can't use modules which use the module object to
>>>> store state. This is commonly used by template libraries to store
>>>> loaded templates in a cache, or 'memoize' expensive functions.
>>>>
>>>> 2. Circular dependencies blow the stack - Its not possible to require
>>>> module A from module B, if module B also requires module A. This
>>>> happens more often than you might think, and is handled by other
>>>> require() implementations by setting the cached module to an empty
>>>> object before eval'ing it. The fix for this requires a module cache to
>>>> be in place.
>>>>
>>>>
>>>> Since these are really hindering progress, I've forked on github and
>>>> committed my proposed patches with associated tests:
>>>> https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
>>>>
>>>> Please, if you have time, review the code and provide me with your feedback.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Caolan
>>>>
>>>
>>> The circular dependency section looks good.
>>>
>>> The bit on caching and testing that things are cached is not good on
>>> the other hand. The way that JS processes are used you can never be
>>> sure if it'll be the same os process handling the request. In the test
>>> suite, its more than likely to be the same OS process because of how
>>> the server gets restarted often and there's a single serialized
>>> client.
>>>
>>> I'm a bit iffy on whether we should cache modules because that'd be a
>>> pretty easy place to break view updates and could lead to other weird
>>> bits in the show/list stuff. Though I also understand the concern for
>>> avoiding all the recompilation. I wonder, is it possible to freeze the
>>> module maybe?
>>>
>>
>
Looks good on first pass, though the resetModuleCache() might be
better off being put into the handler for the reset command.
Re: CommonJS module updates
Posted by Caolan McMahon <ca...@gmail.com>.
An alternative, which avoids caching modules between requests:
https://github.com/caolan/couchdb/commit/76c4f8ef9ab719b3d5cae22255197d06d1395756
I'd prefer if we did cache modules between requests, but this would
still solve my problems ;)
On 22 February 2011 17:20, Caolan McMahon <ca...@gmail.com> wrote:
> I can see the case for not wanting to store state on modules between
> requests, as this could break caching rules for those resources. Even
> though there is a performance hit. How would you know if the results
> for a show or list function had changed?
>
> It also sounds like this would be unreliable anyway since there are
> multiple JS processes. However, I think we must have caching between
> require()'s within a single request, as otherwise some modules will
> not work and we can't handle circular dependencies.
>
> The question is, should CouchDB enforce the fact that applications
> should not store state between requests (except for caching) and take
> the performance hit, or should this be left to commonjs module
> developers?
>
> I assume freezing modules would cancel out the benefits of increased
> module compatability...
>
>
> On 22 February 2011 16:57, Paul Davis <pa...@gmail.com> wrote:
>> On Tue, Feb 22, 2011 at 11:47 AM, Caolan McMahon
>> <ca...@gmail.com> wrote:
>>> As some of you may know, I've been working on a couchapp framework
>>> which makes heavy use of commonjs modules (http://kansojs.org). While
>>> developing this I've run into a number of issues which prevent the use
>>> of some modules, and makes writing my own more difficult:
>>>
>>> 1. Modules are not cached - eval'ing a complex application, consisting
>>> of many modules on each request would have an impact on performance.
>>> It also means you can't use modules which use the module object to
>>> store state. This is commonly used by template libraries to store
>>> loaded templates in a cache, or 'memoize' expensive functions.
>>>
>>> 2. Circular dependencies blow the stack - Its not possible to require
>>> module A from module B, if module B also requires module A. This
>>> happens more often than you might think, and is handled by other
>>> require() implementations by setting the cached module to an empty
>>> object before eval'ing it. The fix for this requires a module cache to
>>> be in place.
>>>
>>>
>>> Since these are really hindering progress, I've forked on github and
>>> committed my proposed patches with associated tests:
>>> https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
>>>
>>> Please, if you have time, review the code and provide me with your feedback.
>>>
>>>
>>> Thanks,
>>>
>>> Caolan
>>>
>>
>> The circular dependency section looks good.
>>
>> The bit on caching and testing that things are cached is not good on
>> the other hand. The way that JS processes are used you can never be
>> sure if it'll be the same os process handling the request. In the test
>> suite, its more than likely to be the same OS process because of how
>> the server gets restarted often and there's a single serialized
>> client.
>>
>> I'm a bit iffy on whether we should cache modules because that'd be a
>> pretty easy place to break view updates and could lead to other weird
>> bits in the show/list stuff. Though I also understand the concern for
>> avoiding all the recompilation. I wonder, is it possible to freeze the
>> module maybe?
>>
>
Re: CommonJS module updates
Posted by Caolan McMahon <ca...@gmail.com>.
I can see the case for not wanting to store state on modules between
requests, as this could break caching rules for those resources. Even
though there is a performance hit. How would you know if the results
for a show or list function had changed?
It also sounds like this would be unreliable anyway since there are
multiple JS processes. However, I think we must have caching between
require()'s within a single request, as otherwise some modules will
not work and we can't handle circular dependencies.
The question is, should CouchDB enforce the fact that applications
should not store state between requests (except for caching) and take
the performance hit, or should this be left to commonjs module
developers?
I assume freezing modules would cancel out the benefits of increased
module compatability...
On 22 February 2011 16:57, Paul Davis <pa...@gmail.com> wrote:
> On Tue, Feb 22, 2011 at 11:47 AM, Caolan McMahon
> <ca...@gmail.com> wrote:
>> As some of you may know, I've been working on a couchapp framework
>> which makes heavy use of commonjs modules (http://kansojs.org). While
>> developing this I've run into a number of issues which prevent the use
>> of some modules, and makes writing my own more difficult:
>>
>> 1. Modules are not cached - eval'ing a complex application, consisting
>> of many modules on each request would have an impact on performance.
>> It also means you can't use modules which use the module object to
>> store state. This is commonly used by template libraries to store
>> loaded templates in a cache, or 'memoize' expensive functions.
>>
>> 2. Circular dependencies blow the stack - Its not possible to require
>> module A from module B, if module B also requires module A. This
>> happens more often than you might think, and is handled by other
>> require() implementations by setting the cached module to an empty
>> object before eval'ing it. The fix for this requires a module cache to
>> be in place.
>>
>>
>> Since these are really hindering progress, I've forked on github and
>> committed my proposed patches with associated tests:
>> https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
>>
>> Please, if you have time, review the code and provide me with your feedback.
>>
>>
>> Thanks,
>>
>> Caolan
>>
>
> The circular dependency section looks good.
>
> The bit on caching and testing that things are cached is not good on
> the other hand. The way that JS processes are used you can never be
> sure if it'll be the same os process handling the request. In the test
> suite, its more than likely to be the same OS process because of how
> the server gets restarted often and there's a single serialized
> client.
>
> I'm a bit iffy on whether we should cache modules because that'd be a
> pretty easy place to break view updates and could lead to other weird
> bits in the show/list stuff. Though I also understand the concern for
> avoiding all the recompilation. I wonder, is it possible to freeze the
> module maybe?
>
Re: CommonJS module updates
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Feb 22, 2011 at 11:47 AM, Caolan McMahon
<ca...@gmail.com> wrote:
> As some of you may know, I've been working on a couchapp framework
> which makes heavy use of commonjs modules (http://kansojs.org). While
> developing this I've run into a number of issues which prevent the use
> of some modules, and makes writing my own more difficult:
>
> 1. Modules are not cached - eval'ing a complex application, consisting
> of many modules on each request would have an impact on performance.
> It also means you can't use modules which use the module object to
> store state. This is commonly used by template libraries to store
> loaded templates in a cache, or 'memoize' expensive functions.
>
> 2. Circular dependencies blow the stack - Its not possible to require
> module A from module B, if module B also requires module A. This
> happens more often than you might think, and is handled by other
> require() implementations by setting the cached module to an empty
> object before eval'ing it. The fix for this requires a module cache to
> be in place.
>
>
> Since these are really hindering progress, I've forked on github and
> committed my proposed patches with associated tests:
> https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
>
> Please, if you have time, review the code and provide me with your feedback.
>
>
> Thanks,
>
> Caolan
>
The circular dependency section looks good.
The bit on caching and testing that things are cached is not good on
the other hand. The way that JS processes are used you can never be
sure if it'll be the same os process handling the request. In the test
suite, its more than likely to be the same OS process because of how
the server gets restarted often and there's a single serialized
client.
I'm a bit iffy on whether we should cache modules because that'd be a
pretty easy place to break view updates and could lead to other weird
bits in the show/list stuff. Though I also understand the concern for
avoiding all the recompilation. I wonder, is it possible to freeze the
module maybe?
Re: CommonJS module updates
Posted by Filipe David Manana <fd...@apache.org>.
For the caching there's already a Jira ticket somewhere.
Can you attach the patch there?
On Tue, Feb 22, 2011 at 4:47 PM, Caolan McMahon
<ca...@gmail.com> wrote:
> As some of you may know, I've been working on a couchapp framework
> which makes heavy use of commonjs modules (http://kansojs.org). While
> developing this I've run into a number of issues which prevent the use
> of some modules, and makes writing my own more difficult:
>
> 1. Modules are not cached - eval'ing a complex application, consisting
> of many modules on each request would have an impact on performance.
> It also means you can't use modules which use the module object to
> store state. This is commonly used by template libraries to store
> loaded templates in a cache, or 'memoize' expensive functions.
>
> 2. Circular dependencies blow the stack - Its not possible to require
> module A from module B, if module B also requires module A. This
> happens more often than you might think, and is handled by other
> require() implementations by setting the cached module to an empty
> object before eval'ing it. The fix for this requires a module cache to
> be in place.
>
>
> Since these are really hindering progress, I've forked on github and
> committed my proposed patches with associated tests:
> https://github.com/caolan/couchdb/compare/7f553e82ef...6c66675a23
>
> Please, if you have time, review the code and provide me with your feedback.
>
>
> Thanks,
>
> Caolan
>
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."