You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Miroslav Rodic <ro...@arco011.com> on 2021/10/20 12:56:49 UTC
Off-line sync by exchanging .couch files
For an application, developed 10 years ago as couchapp with CouchDB 1.1.1, an OFF-LINE data exchange (synchronization) is implemented by adding the local .couch file, with randomly generated name, in a .zip file, which is then sent, by email, to another user and then that user extracts and copies the received .couch file into the local CouchDB instance. The imported .couch file is then synchronized with the appropriate local database using replication. All export/import steps, including the final replication, were implemented within the application, so there were no problems with non-IT users understanding how to use it. Since the application is implemented as a couchapp type, the necessary Erlang functions have been developed that allow archiving/copying/etc.
The question follows: is it possible to keep the same approach in off-line synchronization with the latest version of CouchDB, which is 3.2.0, when the installed CouchDB is configured as a single node? If this is not possible, then which solutions are recommended for off-line synchronization?
Copying of .couch files is not clearly explained in the backup documentation. A backup assumes, if I understood correctly, that .couch files can be copied to a secure location, and then, when needed, can be restored by overwriting the existing .couch files in the data directory. If, for any reason, the corresponding DB is deleted by using the Fauxton before the .couch files are reverted back, then reverted .couch files will not be accepted and displayed in Fauxton. This is the behavior I had on my comp testing the backup/restore procedure.
Backup/restore procedure tested on OS X 10.15.7 with CouchDB 3.2.0, where path to data directory is as follows: ~/Library/Application Support/CouchDB2/var/lib/couchdb.
Regards,
Miroslav
Re: Off-line sync by exchanging .couch files
Posted by Jan Lehnardt <ja...@apache.org>.
Hi Miroslav,
that’s a nice application :)
You should be able to do the same thing with CouchDB 3.2.0, but you need to do an extra step.
In CouchDB 1.x, when opening a a database `foo`, it would open a file at `$database_dir/foo.couch`, very straightforward.
In CouchDB 2.x and 3.x, the process has one more step:
1. request to open database `foo`
2. CouchDB looks in its internal database `/_node/_local/dbs` for the document `foo` (/_node/_local/dbs/foo)
- The structure of that document is explained in https://docs.couchdb.org/en/stable/cluster/sharding.html#updating-cluster-metadata-to-reflect-the-new-target-shard-s
3. read the shard suffix and the shard range from the document (a single node database created with q=1 will only have one shard range 00000000-ffffffff).
- the suffix is in bytes, but they are a regular UNIX timestamp, you’ll have to convert it for use on the file system.
4. open this file now: `$database_dir/shards/00000000-ffffffff/foo.$suffix.couch
So to follow your procedure you need to:
1. create a document in the internal _dbs database with the same _id as the database name
2. put the .couch file in the place specified by the shard range and `shards/` subdirectory and match the shard suffix.
Voilá
Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
*24/7 Observation for your CouchDB Instances:
https://opservatory.app
> On 20. Oct 2021, at 14:56, Miroslav Rodic <ro...@arco011.com> wrote:
>
> For an application, developed 10 years ago as couchapp with CouchDB 1.1.1, an OFF-LINE data exchange (synchronization) is implemented by adding the local .couch file, with randomly generated name, in a .zip file, which is then sent, by email, to another user and then that user extracts and copies the received .couch file into the local CouchDB instance. The imported .couch file is then synchronized with the appropriate local database using replication. All export/import steps, including the final replication, were implemented within the application, so there were no problems with non-IT users understanding how to use it. Since the application is implemented as a couchapp type, the necessary Erlang functions have been developed that allow archiving/copying/etc.
>
> The question follows: is it possible to keep the same approach in off-line synchronization with the latest version of CouchDB, which is 3.2.0, when the installed CouchDB is configured as a single node? If this is not possible, then which solutions are recommended for off-line synchronization?
>
> Copying of .couch files is not clearly explained in the backup documentation. A backup assumes, if I understood correctly, that .couch files can be copied to a secure location, and then, when needed, can be restored by overwriting the existing .couch files in the data directory. If, for any reason, the corresponding DB is deleted by using the Fauxton before the .couch files are reverted back, then reverted .couch files will not be accepted and displayed in Fauxton. This is the behavior I had on my comp testing the backup/restore procedure.
>
> Backup/restore procedure tested on OS X 10.15.7 with CouchDB 3.2.0, where path to data directory is as follows: ~/Library/Application Support/CouchDB2/var/lib/couchdb.
>
> Regards,
> Miroslav
>
Re: Off-line sync by exchanging .couch files
Posted by Miroslav Rodic <ro...@arco011.com>.
Dear Jan,
Thank you for the detailed explanation!
Regards,
Miroslav
> On 20 Oct 2021, at 17:19, Jan Lehnardt <ja...@apache.org> wrote:
>
> Oh, and note that for 3.2.0 and CouchApps to work, you will have to change the default CSP headers: https://docs.couchdb.org/en/stable/cve/2021-38295.html / https://github.com/apache/couchdb/pull/3724
>
> Best
> Jan
> —
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
> *24/7 Observation for your CouchDB Instances:
> https://opservatory.app
>
>> On 20. Oct 2021, at 14:56, Miroslav Rodic <ro...@arco011.com> wrote:
>>
>> For an application, developed 10 years ago as couchapp with CouchDB 1.1.1, an OFF-LINE data exchange (synchronization) is implemented by adding the local .couch file, with randomly generated name, in a .zip file, which is then sent, by email, to another user and then that user extracts and copies the received .couch file into the local CouchDB instance. The imported .couch file is then synchronized with the appropriate local database using replication. All export/import steps, including the final replication, were implemented within the application, so there were no problems with non-IT users understanding how to use it. Since the application is implemented as a couchapp type, the necessary Erlang functions have been developed that allow archiving/copying/etc.
>>
>> The question follows: is it possible to keep the same approach in off-line synchronization with the latest version of CouchDB, which is 3.2.0, when the installed CouchDB is configured as a single node? If this is not possible, then which solutions are recommended for off-line synchronization?
>>
>> Copying of .couch files is not clearly explained in the backup documentation. A backup assumes, if I understood correctly, that .couch files can be copied to a secure location, and then, when needed, can be restored by overwriting the existing .couch files in the data directory. If, for any reason, the corresponding DB is deleted by using the Fauxton before the .couch files are reverted back, then reverted .couch files will not be accepted and displayed in Fauxton. This is the behavior I had on my comp testing the backup/restore procedure.
>>
>> Backup/restore procedure tested on OS X 10.15.7 with CouchDB 3.2.0, where path to data directory is as follows: ~/Library/Application Support/CouchDB2/var/lib/couchdb.
>>
>> Regards,
>> Miroslav
>>
>
Re: Off-line sync by exchanging .couch files
Posted by Jan Lehnardt <ja...@apache.org>.
Oh, and note that for 3.2.0 and CouchApps to work, you will have to change the default CSP headers: https://docs.couchdb.org/en/stable/cve/2021-38295.html / https://github.com/apache/couchdb/pull/3724
Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
*24/7 Observation for your CouchDB Instances:
https://opservatory.app
> On 20. Oct 2021, at 14:56, Miroslav Rodic <ro...@arco011.com> wrote:
>
> For an application, developed 10 years ago as couchapp with CouchDB 1.1.1, an OFF-LINE data exchange (synchronization) is implemented by adding the local .couch file, with randomly generated name, in a .zip file, which is then sent, by email, to another user and then that user extracts and copies the received .couch file into the local CouchDB instance. The imported .couch file is then synchronized with the appropriate local database using replication. All export/import steps, including the final replication, were implemented within the application, so there were no problems with non-IT users understanding how to use it. Since the application is implemented as a couchapp type, the necessary Erlang functions have been developed that allow archiving/copying/etc.
>
> The question follows: is it possible to keep the same approach in off-line synchronization with the latest version of CouchDB, which is 3.2.0, when the installed CouchDB is configured as a single node? If this is not possible, then which solutions are recommended for off-line synchronization?
>
> Copying of .couch files is not clearly explained in the backup documentation. A backup assumes, if I understood correctly, that .couch files can be copied to a secure location, and then, when needed, can be restored by overwriting the existing .couch files in the data directory. If, for any reason, the corresponding DB is deleted by using the Fauxton before the .couch files are reverted back, then reverted .couch files will not be accepted and displayed in Fauxton. This is the behavior I had on my comp testing the backup/restore procedure.
>
> Backup/restore procedure tested on OS X 10.15.7 with CouchDB 3.2.0, where path to data directory is as follows: ~/Library/Application Support/CouchDB2/var/lib/couchdb.
>
> Regards,
> Miroslav
>