npm and couchdb

[Benoit Chesneau <bc...@gmail.com>]

Some people may have noticed it but npm inc is now live for real and
has raised $ 2.5 M [1]. It may be a good time to communicate that
couchdb even behind a layer is in used in npm. Also maybe Jason has
some interesting stories to give. Maybe one that know how to write can
take it ;)

I also found that this  other post  in their blog [2] can be
interesting for other couchdb users and dev.  Not saying that novacut
is taking a similar approach though different (not using any private
layer) [3] in demedia.


Just saying.

- benoit

[1] http://blog.npmjs.org/post/76320673650/funding
[2] http://blog.npmjs.org/post/75707294465/new-npm-registry-architecture
[3] https://launchpad.net/dmedia

Re: npm and couchdb

[Benoit Chesneau <bc...@gmail.com>]

On Wed, Feb 12, 2014 at 12:43 PM, Robert Samuel Newson
<rn...@apache.org> wrote:
>
> I can't think of a way to say that couchdb is still used but behind a layer to work around perceived deficiencies, so I would personally say nothing.
>

I was thinking that even if the story isn't glorious for us, we should
still be able to say that npm success wouldn't be here without
couchbd. At least we can say it. The other things is to profit from
this occasion to tell to everyone that we are (some are) actually
working on improving the situation and explain the why, what is really
wrong, what could be improved. That would actually be a good thing
showing openly our failures. Also maybe someone would came with an
unexpected solution.

> The story is deeper and wider than I know but I do know that at least some of the issues encountered here are to do with the approach taken, rather than couchdb being entirely culpable. Mikeal posted recently about the history of the architecture here and seems to confirm that. Had something like the global changes feed existed at the time and a database-per-package approach been taken, I think the story would have been radically different.

true

>
> I would rather focus on the actual problems in couchdb rather than the problems in the original npm architectures use of npm; that plane has sailed.
>
> The main problems are around the replicator. It's a complicated piece of code (and without technical justification for that complexity, it's our failure to simplify) and can fail to make progress under some conditions (notably, and pertinently, attachment heavy databases). The replicator, for all its complexity, does not make some obvious optimizations. A partially transferred attachment followed by a connection break causes the full re-transfer of the attachment. This can be a real killer and I believe was a real issue in npm registry replication. If a document is updated, all the attachments are replicated again, even if they exist on the target.
>
> CouchDB *ought* to have been the perfect solution to the problem of mirroring a repository such as npm. We have made significant improvements to the replicator in the last several releases, all of which made npm replication better. That work should continue, and I think it's time for a new replicator implementation; we can do better.
>
> A further improvement, that Volker has certainly attempted recently with some (but not total) success, would be to allow replication to work from a static host server. Currently, replication requires couchdb, or couchdb-like system, at source and target. That enables some important optimizations but if it were not mandatory it would open up some interesting possibilities.

+1
>
> B.
>
> On 12 Feb 2014, at 11:12, Benoit Chesneau <bc...@gmail.com> wrote:
>
>> Some people may have noticed it but npm inc is now live for real and
>> has raised $ 2.5 M [1]. It may be a good time to communicate that
>> couchdb even behind a layer is in used in npm. Also maybe Jason has
>> some interesting stories to give. Maybe one that know how to write can
>> take it ;)
>>
>> I also found that this  other post  in their blog [2] can be
>> interesting for other couchdb users and dev.  Not saying that novacut
>> is taking a similar approach though different (not using any private
>> layer) [3] in demedia.
>>
>>
>> Just saying.
>>
>> - benoit
>>
>> [1] http://blog.npmjs.org/post/76320673650/funding
>> [2] http://blog.npmjs.org/post/75707294465/new-npm-registry-architecture
>> [3] https://launchpad.net/dmedia
>

Re: npm and couchdb

[Robert Samuel Newson <rn...@apache.org>]

I can’t think of a way to say that couchdb is still used but behind a layer to work around perceived deficiencies, so I would personally say nothing.

The story is deeper and wider than I know but I do know that at least some of the issues encountered here are to do with the approach taken, rather than couchdb being entirely culpable. Mikeal posted recently about the history of the architecture here and seems to confirm that. Had something like the global changes feed existed at the time and a database-per-package approach been taken, I think the story would have been radically different.

I would rather focus on the actual problems in couchdb rather than the problems in the original npm architectures use of npm; that plane has sailed.

The main problems are around the replicator. It’s a complicated piece of code (and without technical justification for that complexity, it’s our failure to simplify) and can fail to make progress under some conditions (notably, and pertinently, attachment heavy databases). The replicator, for all its complexity, does not make some obvious optimizations. A partially transferred attachment followed by a connection break causes the full re-transfer of the attachment. This can be a real killer and I believe was a real issue in npm registry replication. If a document is updated, all the attachments are replicated again, even if they exist on the target.

CouchDB *ought* to have been the perfect solution to the problem of mirroring a repository such as npm. We have made significant improvements to the replicator in the last several releases, all of which made npm replication better. That work should continue, and I think it’s time for a new replicator implementation; we can do better.

A further improvement, that Volker has certainly attempted recently with some (but not total) success, would be to allow replication to work from a static host server. Currently, replication requires couchdb, or couchdb-like system, at source and target. That enables some important optimizations but if it were not mandatory it would open up some interesting possibilities.

B.

On 12 Feb 2014, at 11:12, Benoit Chesneau <bc...@gmail.com> wrote:

> Some people may have noticed it but npm inc is now live for real and
> has raised $ 2.5 M [1]. It may be a good time to communicate that
> couchdb even behind a layer is in used in npm. Also maybe Jason has
> some interesting stories to give. Maybe one that know how to write can
> take it ;)
> 
> I also found that this  other post  in their blog [2] can be
> interesting for other couchdb users and dev.  Not saying that novacut
> is taking a similar approach though different (not using any private
> layer) [3] in demedia.
> 
> 
> Just saying.
> 
> - benoit
> 
> [1] http://blog.npmjs.org/post/76320673650/funding
> [2] http://blog.npmjs.org/post/75707294465/new-npm-registry-architecture
> [3] https://launchpad.net/dmedia