Adding vs replicating documents

[Luca Matteis <lm...@gmail.com>]

I have a file that contains lots of JSON documents - probably around
1gb of data.
I uploaded this file to my server's Couch instance (on the cloud)
using the Bulk API.
Now I need to re-upload it because I've made some changes to the file
- I've added some documents, deleted some, and edited a few.

My question is, can I avoid sending over the wire the 1gb of data
every time I make modifications to it?

I've thought about setting up a local Couch instance on my computer,
upload data there, and then replicate it to my Couch instance, so that
only changes get sent over the wire... but was wondering if there was
an easier way instead of having to install CouchDB on my computer.

Re: Adding vs replicating documents

[Jens Alfke <je...@couchbase.com>]

On Jun 7, 2012, at 8:12 PM, Luca Matteis wrote:

> Ok basically I have a program that runs on my local computer that
> generates data that I want to insert/modify to my Couch instance. So
> my local data should be in synch with my cloud data... but i can't
> install CouchDB locally... so how do I keep it in synch with my hosted
> Couch instance without having to re-upload all of my data every time?

For each record in the local file, decide whether it's different from what's in CouchDB. If it is, PUT the document to the database.

How to tell whether it's different depends on what's in your file. If you have timestamps it's pretty easy. Otherwise you may have to do something like storing a local side file with checksums of every document, and then comparing the values in the file against the saved checksums. The checksum file can also store the rev ID from the last time you saved to CouchDB, which will make saving easier.

—Jens

Re: Adding vs replicating documents

[Tim Tisdall <ti...@gmail.com>]

I was going to suggest something like this too.  However, if you have
access to the script, you could alter it to record in a separate file
which documents have been altered and then write another script to
only send those docs to the couchdb server.  (This is essentially
duplicating what the couchdb replication would do, but without having
to have couchdb on your local machine)  A really simple way is to just
output the doc ids you've modified to a text file and then wipe that
file out after you've sent the updates to the server.

On Fri, Jun 8, 2012 at 2:10 AM, Matthieu Rakotojaona
<ma...@gmail.com> wrote:
> I guess you can write some script that makes a diff between the old
> version and the new version, and returns the changed docs. You could
> then upload only those changed docs.
>
> --
> Matthieu RAKOTOJAONA

Re: Adding vs replicating documents

[Matthieu Rakotojaona <ma...@gmail.com>]

I guess you can write some script that makes a diff between the old
version and the new version, and returns the changed docs. You could
then upload only those changed docs.

-- 
Matthieu RAKOTOJAONA

Re: Adding vs replicating documents

[Dave Cottlehuber <da...@muse.net.nz>]

On 8 June 2012 06:26, Luca Matteis <lm...@gmail.com> wrote:

> Ok, say I install a local Couch that I upload my 1gb of data to.
> Whenever a change is made to a JSON document in my file, my routine
> will still have to go through all the records and update my local
> Couch, which would result in the entire DB updating, therefore the
> replication would need to transfer *everything* to my hosted Couch
> once again.
>
>
Continuous replication would be an obvious solution to this.

If you need to intervene programmatically then sit on the _changes feed,
keep track of the ids that change and then specify the URLs to replicate
manually. This is covered on the wiki & in the guide also, you have the
links
by now  :-)

If your JSON docs are generated by script with ids, then you can simply
regenerate them into your local couch (no need for replication) and
the revs etc will all match up nicely with the remote end.

Key misconceptions:

#1 changes feed will allow you to find out changes since a given DB update.

#2 replication does not transfer all documents, it compares docs and
missing revs
and makes smart decisions on what needs to be transferred. If a doc exists
in the source but not in the destination, the whole of that doc *does* of
course
need to be transferred. That's likely a simplistic summary but will do for
the
moment.

A+
Dave

Re: Adding vs replicating documents

[Luca Matteis <lm...@gmail.com>]

Ok, say I install a local Couch that I upload my 1gb of data to.
Whenever a change is made to a JSON document in my file, my routine
will still have to go through all the records and update my local
Couch, which would result in the entire DB updating, therefore the
replication would need to transfer *everything* to my hosted Couch
once again.

Maybe an easy way around this would be to find what files changed in
my JSON file, by simply looping through it.

Because I don't think a local Couch will solve the issue - or am I
missing something?

On Fri, Jun 8, 2012 at 6:15 AM, Mark Hahn <ma...@hahnca.com> wrote:
>>   so how do I keep it in synch with my hosted Couch instance
>
> The obvious solution is to either manually use futon or to have a local
> copy of couchdb and use the built-in replication.  So your program on your
> local computer could either generate data you could manually update or it
> could update your local couchdb.  I've already asked about futon.  So let
> me ask, why can't you run a local couchdb?  It runs on windows, macs, and
> linux.
>
> To answer you original question, no, you cannot update over the net without
> some way to link docs.  I can't imagine how any DB could do this.
>
> On Thu, Jun 7, 2012 at 8:12 PM, Luca Matteis <lm...@gmail.com> wrote:
>
>> Because my JSON file is generated programmatically.
>>
>> Ok basically I have a program that runs on my local computer that
>> generates data that I want to insert/modify to my Couch instance. So
>> my local data should be in synch with my cloud data... but i can't
>> install CouchDB locally... so how do I keep it in synch with my hosted
>> Couch instance without having to re-upload all of my data every time?
>>
>> On Fri, Jun 8, 2012 at 4:46 AM, Mark Hahn <ma...@hahnca.com> wrote:
>> >>   I can't use futon.
>> >
>> > Why not?
>>

Re: Adding vs replicating documents

[Mark Hahn <ma...@hahnca.com>]

>   so how do I keep it in synch with my hosted Couch instance

The obvious solution is to either manually use futon or to have a local
copy of couchdb and use the built-in replication.  So your program on your
local computer could either generate data you could manually update or it
could update your local couchdb.  I've already asked about futon.  So let
me ask, why can't you run a local couchdb?  It runs on windows, macs, and
linux.

To answer you original question, no, you cannot update over the net without
some way to link docs.  I can't imagine how any DB could do this.

On Thu, Jun 7, 2012 at 8:12 PM, Luca Matteis <lm...@gmail.com> wrote:

> Because my JSON file is generated programmatically.
>
> Ok basically I have a program that runs on my local computer that
> generates data that I want to insert/modify to my Couch instance. So
> my local data should be in synch with my cloud data... but i can't
> install CouchDB locally... so how do I keep it in synch with my hosted
> Couch instance without having to re-upload all of my data every time?
>
> On Fri, Jun 8, 2012 at 4:46 AM, Mark Hahn <ma...@hahnca.com> wrote:
> >>   I can't use futon.
> >
> > Why not?
>

Re: Adding vs replicating documents

[Luca Matteis <lm...@gmail.com>]

Because my JSON file is generated programmatically.

Ok basically I have a program that runs on my local computer that
generates data that I want to insert/modify to my Couch instance. So
my local data should be in synch with my cloud data... but i can't
install CouchDB locally... so how do I keep it in synch with my hosted
Couch instance without having to re-upload all of my data every time?

On Fri, Jun 8, 2012 at 4:46 AM, Mark Hahn <ma...@hahnca.com> wrote:
>>   I can't use futon.
>
> Why not?

Re: Adding vs replicating documents

[Luca Matteis <lm...@gmail.com>]

Because my JSON file is generated programmatically.

Ok basically I have a program that runs on my local computer that
generates data that I want to insert/modify to my Couch instance. So
my local data should be in synch with my cloud data... but i can't
install CouchDB locally... so how do I keep it in synch with my hosted
Couch instance without having to re-upload all of my data every time?

On Fri, Jun 8, 2012 at 4:46 AM, Mark Hahn <ma...@hahnca.com> wrote:
>>   I can't use futon.
>
> Why not?

Re: Adding vs replicating documents

[Mark Hahn <ma...@hahnca.com>]

>   I can't use futon.

Why not?

Re: Adding vs replicating documents

[Luca Matteis <lm...@gmail.com>]

On Fri, Jun 8, 2012 at 4:35 AM, Simon Metson <si...@cloudant.com> wrote:
> Hi,
> Can't your program write to a local CouchDB instance instead of a file?

Right, the thing is that I can't install CouchDB on my local computer.
That's why I'm wondering if there's a simpler solution, but probably
not, right?

Re: Adding vs replicating documents

[Simon Metson <si...@cloudant.com>]

Hi, 
Can't your program write to a local CouchDB instance instead of a file?
Cheers
Simon 


On Friday, 8 June 2012 at 03:31, Luca Matteis wrote:

> I use my own IDs which are stored in the JSON file. I can't use futon.
> I'm stuck with this JSON file (which is generated programmatically
> btw). I was just wondering if there was an easy way to use the
> replication API and check if this JSON file has changes, and upload
> only those, instead of re-uploading everything.
> 
> On Fri, Jun 8, 2012 at 4:14 AM, Mark Hahn <mark@hahnca.com (mailto:mark@hahnca.com)> wrote:
> > You can't update a document unless you have the id.  Do you have a view set
> > up to find what you want?  If so pull them up and edit them in futon.
> > 
> > Usually one would edit the docs in the db directly using futon, not change
> > the original data.
> > 
> > On Thu, Jun 7, 2012 at 6:35 PM, Luca Matteis <lmatteis@gmail.com (mailto:lmatteis@gmail.com)> wrote:
> > 
> > > I have a file that contains lots of JSON documents - probably around
> > > 1gb of data.
> > > I uploaded this file to my server's Couch instance (on the cloud)
> > > using the Bulk API.
> > > Now I need to re-upload it because I've made some changes to the file
> > > - I've added some documents, deleted some, and edited a few.
> > > 
> > > My question is, can I avoid sending over the wire the 1gb of data
> > > every time I make modifications to it?
> > > 
> > > I've thought about setting up a local Couch instance on my computer,
> > > upload data there, and then replicate it to my Couch instance, so that
> > > only changes get sent over the wire... but was wondering if there was
> > > an easier way instead of having to install CouchDB on my computer.
> > > 
> > 
> > 
> 
> 
> 

Re: Adding vs replicating documents

[Luca Matteis <lm...@gmail.com>]

I use my own IDs which are stored in the JSON file. I can't use futon.
I'm stuck with this JSON file (which is generated programmatically
btw). I was just wondering if there was an easy way to use the
replication API and check if this JSON file has changes, and upload
only those, instead of re-uploading everything.

On Fri, Jun 8, 2012 at 4:14 AM, Mark Hahn <ma...@hahnca.com> wrote:
> You can't update a document unless you have the id.  Do you have a view set
> up to find what you want?  If so pull them up and edit them in futon.
>
> Usually one would edit the docs in the db directly using futon, not change
> the original data.
>
> On Thu, Jun 7, 2012 at 6:35 PM, Luca Matteis <lm...@gmail.com> wrote:
>
>> I have a file that contains lots of JSON documents - probably around
>> 1gb of data.
>> I uploaded this file to my server's Couch instance (on the cloud)
>> using the Bulk API.
>> Now I need to re-upload it because I've made some changes to the file
>> - I've added some documents, deleted some, and edited a few.
>>
>> My question is, can I avoid sending over the wire the 1gb of data
>> every time I make modifications to it?
>>
>> I've thought about setting up a local Couch instance on my computer,
>> upload data there, and then replicate it to my Couch instance, so that
>> only changes get sent over the wire... but was wondering if there was
>> an easier way instead of having to install CouchDB on my computer.
>>

Re: Adding vs replicating documents

[Mark Hahn <ma...@hahnca.com>]

You can't update a document unless you have the id.  Do you have a view set
up to find what you want?  If so pull them up and edit them in futon.

Usually one would edit the docs in the db directly using futon, not change
the original data.

On Thu, Jun 7, 2012 at 6:35 PM, Luca Matteis <lm...@gmail.com> wrote:

> I have a file that contains lots of JSON documents - probably around
> 1gb of data.
> I uploaded this file to my server's Couch instance (on the cloud)
> using the Bulk API.
> Now I need to re-upload it because I've made some changes to the file
> - I've added some documents, deleted some, and edited a few.
>
> My question is, can I avoid sending over the wire the 1gb of data
> every time I make modifications to it?
>
> I've thought about setting up a local Couch instance on my computer,
> upload data there, and then replicate it to my Couch instance, so that
> only changes get sent over the wire... but was wondering if there was
> an easier way instead of having to install CouchDB on my computer.
>