You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by andy e <vi...@gmail.com> on 2013/05/14 05:30:39 UTC
Having trouble replicating with a large couch instance (npm repo)
Hi,
Based on the awesome feedback I got from the nodejs list (see
https://groups.google.com/d/msg/nodejs/sX4mbsRPwls/WtDDE-To2o4J), we tried
replicating with their couch instance so we could use it in an offline
environment.
We're essentially following the instructions at
http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository but
replication fails after syncing about 17k documents.
We've tried reinstalling couch (found one issue that suggested using a
patched version of SpiderMonkey) but the same thing keeps happening, even
after restarting replication several times.
Here's our setup:
CentOS 6.4
CouchDB 1.3
SpiderMonkey 1.8.5-7
Replication works fine for over 17,000 documents, then we see this error
and can't get past it:
[Sat, 11 May 2013 00:55:39 GMT] [error] [<0.12970.4>] Replicator: couldn't
write document `bufferhelper`, revision
`19-d339684ee7f5eaf4cc18d84da753832d`, to target database `registry`.
Error: `unauthorized`, reason: `Please log in before writing to the db`.
Any ideas?
Thanks,
Andy
Re: Having trouble replicating with a large couch instance (npm repo)
Posted by andy e <vi...@gmail.com>.
Dave,
Thanks for the tips/feedback!
On Tue, May 14, 2013 at 3:33 AM, Dave Cottlehuber <dc...@jsonified.com> wrote:
> On 14 May 2013 05:30, andy e <vi...@gmail.com> wrote:
>
> > Hi,
>
>
>
> Based on the awesome feedback I got from the nodejs list (see
>
> https://groups.google.com/d/msg/nodejs/sX4mbsRPwls/WtDDE-To2o4J), we tried
>
> replicating with their couch instance so we could use it in an offline
>
> environment.
>
>
>
> We're essentially following the instructions at
>
> http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository but
>
> replication fails after syncing about 17k documents.
>
>
>
> We've tried reinstalling couch (found one issue that suggested using a
>
> patched version of SpiderMonkey) but the same thing keeps happening, even
>
> after restarting replication several times.
>
>
>
> Here's our setup:
>
>
>
> CentOS 6.4
>
> CouchDB 1.3
>
> SpiderMonkey 1.8.5-7
>
>
>
> Replication works fine for over 17,000 documents, then we see this error
>
> and can't get past it:
>
>
>
> [Sat, 11 May 2013 00:55:39 GMT] [error] [<0.12970.4>] Replicator: couldn't
>
> write document `bufferhelper`, revision
>
> `19-d339684ee7f5eaf4cc18d84da753832d`, to target database `registry`.
>
> Error: `unauthorized`, reason: `Please log in before writing to the db`.
>
>
>
> Any ideas?
>
>
>
> Thanks,
>
>
>
> Andy
>
>
>
> Hmm, weird.
> curl -svX GET http://isaacs.iriscouch.com/registry/bufferhelper |
json_reformat looks ok to me, as does:
> curl -svX GET
http://registry.npmjs.org/bufferhelper/-/bufferhelper-0.2.0.tgz
| tar tzf -
> && echo OK
> And my local npm mirror is up to date, so let's assume that the issue is at
your end for the moment.
> You can replicate single (or multiple) specific documents, instead of the
whole npm repo, using this, where doc_ids is an array of doc ids you want
to replicate:
> curl -svX POST http://admin:passwd@127.0.0.1:5984/_replicate -d
> '{"source":"
http://isaacs.iriscouch.com/registry/
", "target":"mirror",
> "create_target":true, "doc_ids":["bufferhelper"]}'
-HContent-Type:application/json | json_reformat
> And if that fails, can you post the output + anything of interest in
couch.log please, along with how are you setting up your replication? e.g.
curl …. as above.
We didn't have an admin user set up, so we added that and tried basically
this command and it seemed to go past 17k, so maybe that was the trick.
>
> Some tips:
> If you're using the replicator db (via POST _replicator endpoint), I'd
suggest ensuring you have appropriate permissions in the request, e.g.
> curl -svX PUT http://admin:passwd@127.0.0.1:5984/_replicator/mirror_npm -d
'{"source":"http://isaacs.iriscouch.com/registry/", "target":"mirror",
"user_ctx": {"name": "admin"}, "doc_ids":["bufferhelper"]}'
-HContent-Type:application/json
>
> You can obviously add "create_target":true, and "continuous": true as
needed as usual.
> In your case of needing periodic updates, I'd not use continuous
replication, as the normal one will return from the API call only when
finished successfully, or failed.
> You may well find that https://npmjs.org/package/npm-gh or
http://debuggable.com/posts/private-npm-modules:4e68cc7d-1ac4-42d9-995a-343dcbdd56cbis
a more suitable option.
I wish we could use this, but we need to be able to install 'public'
modules (say express, which has dependencies on other stuff obviously) in
addition to our own. But, we can't access the npm repo (or github) from
where we need to do this, hence wanting a local copy. Kind of stinks but
it's life for us at the moment.
> For publishing modules to different registries, consider doing alias
mynpm='npm --userconfig /path/to/other/npmrc --registry
https://url/for/other/registry
' from
> https://github.com/isaacs/npm/issues/2593
> A+
Dave
Re: Having trouble replicating with a large couch instance (npm repo)
Posted by Dave Cottlehuber <dc...@jsonified.com>.
On 14 May 2013 05:30, andy e <vi...@gmail.com> wrote:
> Hi,
>
> Based on the awesome feedback I got from the nodejs list (see
> https://groups.google.com/d/msg/nodejs/sX4mbsRPwls/WtDDE-To2o4J), we tried
> replicating with their couch instance so we could use it in an offline
> environment.
>
> We're essentially following the instructions at
> http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository but
> replication fails after syncing about 17k documents.
>
> We've tried reinstalling couch (found one issue that suggested using a
> patched version of SpiderMonkey) but the same thing keeps happening, even
> after restarting replication several times.
>
> Here's our setup:
>
> CentOS 6.4
> CouchDB 1.3
> SpiderMonkey 1.8.5-7
>
> Replication works fine for over 17,000 documents, then we see this error
> and can't get past it:
>
> [Sat, 11 May 2013 00:55:39 GMT] [error] [<0.12970.4>] Replicator: couldn't
> write document `bufferhelper`, revision
> `19-d339684ee7f5eaf4cc18d84da753832d`, to target database `registry`.
> Error: `unauthorized`, reason: `Please log in before writing to the db`.
>
> Any ideas?
>
> Thanks,
>
> Andy
>
Hmm, weird.
curl -svX GET http://isaacs.iriscouch.com/registry/bufferhelper |
json_reformat looks ok to me, as does:
curl -svX GET
http://registry.npmjs.org/bufferhelper/-/bufferhelper-0.2.0.tgz | tar tzf -
&& echo OK
And my local npm mirror is up to date, so let's assume that the issue is at
your end for the moment.
You can replicate single (or multiple) specific documents, instead of the
whole npm repo, using this, where doc_ids is an array of doc ids you want
to replicate:
curl -svX POST http://admin:passwd@127.0.0.1:5984/_replicate -d '{"source":"
http://isaacs.iriscouch.com/registry/", "target":"mirror",
"create_target":true, "doc_ids":["bufferhelper"]}'
-HContent-Type:application/json | json_reformat
And if that fails, can you post the output + anything of interest in
couch.log please, along with how are you setting up your replication? e.g.
curl …. as above.
Some tips:
If you're using the replicator db (via POST _replicator endpoint), I'd
suggest ensuring you have appropriate permissions in the request, e.g.
curl -svX PUT http://admin:passwd@127.0.0.1:5984/_replicator/mirror_npm -d
'{"source":"http://isaacs.iriscouch.com/registry/", "target":"mirror",
"user_ctx": {"name": "admin"}, "doc_ids":["bufferhelper"]}'
-HContent-Type:application/json
You can obviously add "create_target":true, and "continuous": true as
needed as usual.
In your case of needing periodic updates, I'd not use continuous
replication, as the normal one will return from the API call only when
finished successfully, or failed.
You may well find that https://npmjs.org/package/npm-gh or
http://debuggable.com/posts/private-npm-modules:4e68cc7d-1ac4-42d9-995a-343dcbdd56cbis
a more suitable option.
For publishing modules to different registries, consider doing alias
mynpm='npm --userconfig /path/to/other/npmrc --registry
https://url/for/other/registry' from
https://github.com/isaacs/npm/issues/2593
A+
Dave