You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Ram Rachum <ra...@rachum.com> on 2015/06/12 12:15:34 UTC
CouchDB questions
Hi everyone,
I've never used CouchDB before, and I'm considering using it for a new
project of mine. I have a couple of questions about it. (Note that I'm a
database noob in general.)
I'm thinking of making a desktop app that syncs with a Django app. I want
the Django app to have a database, and the desktop app to have one too,
which would have the same data. The hard part is that I want to be able to
work with the desktop app while offline, so I'll be writing changes to the
database, and then later when it's online it'll sync the two databases
together.
Is that something that CouchDB can handle?
If so, two questions:
1. What happens on merge fails? (i.e. when the two masters wrote data that
contradicts each other.) I'd want to have a manual resolve, but what would
be the interface to resolving it?
2. I'm assuming that CouchDB works by keeping a log of all transactions
made to the database. I want to provide undo functionality to the program
using this log of transactions. (i.e. every undo would look at the last
transaction and create a transaction that's the reverse of it.) This means
I need each transaction to save the old data, and I need the possibility to
access the transaction table. Is this possible?
Thanks for your help,
Ram.
Re: CouchDB utc_id behavior
Posted by Adam Kocoloski <ko...@apache.org>.
Awesome, thanks for replying back to the thread Kiril. Cheers,
Adam
> On Jun 18, 2015, at 1:14 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>
> Solved.
> If Couch starts ~ 30 sec. after boot completion, the problem seems to go away.
> Thanks all.
> ------------------------------------------------------------------------
> *With best regards,*
> Kiril Stankov
>
> On 18-Jun-15 2:15 PM, Kiril Stankov wrote:
>> Yes, the one from 12:59 is Ok. The one from 12:51 is not.
>> I will try now to delay the start of Couchdb with a minute after all other daemons started and see if it will help.
>> May be the hardware clock is "off" and it takes time until ntpd syncs it and CouchDB starts in between.
>>
>>
>> ------------------------------------------------------------------------
>> *With best regards,*
>> Kiril Stankov,
>>
>> On 17-Jun-15 6:41 PM, Adam Kocoloski wrote:
>>> Do you happen to know if either one of these was correct-ish? Do you see any timestamps in the access logs that are also “off”?
>>>
>>> 05188de92ef02f -> Mon, 15 Jun 2015 12:51:05 GMT
>>> 05188e0805067f -> Mon, 15 Jun 2015 12:59:42 GMT
>>>
>>> Adam
>>>
>>>> On Jun 16, 2015, at 12:44 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>>>>
>>>> Ubuntu, couch 1.6.1, apt-get update few days ago.
>>>> --
>>>> Regards,
>>>>
>>>> Kiril Stankov,
>>>> OpenNet Software ltd.
>>>>
>>>>
>>>> -------- Original Message --------
>>>> From: Nick North <no...@gmail.com>
>>>> Sent: June 16, 2015 7:41:50 PM GMT+03:00
>>>> To: user@couchdb.apache.org, Joan Touzet <wo...@apache.org>
>>>> Subject: Re: CouchDB utc_id behavior
>>>>
>>>> Thanks for the correction Joan - I had forgotten about the possibility of
>>>> the clock jumping backwards. However, in this case the obvious causes don't
>>>> seem to apply, and I'm slightly at a loss. I can't hypothesise a mechanism
>>>> that would cause now() to return these times if the OS clocks are in sync.
>>>> Kiril - what setup are you running on?
>>>>
>>>> Nick
>>>>
>>>> On Tue, 16 Jun 2015 at 15:48 Kiril Stankov <ki...@open-net.biz> wrote:
>>>>
>>>>> Hi,
>>>>> As I wrote, ntpd is running and both machines have synced time. They were
>>>>> not down for weeks.
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Kiril Stankov,
>>>>> OpenNet Software ltd.
>>>>>
>>>>>
>>>>> -------- Original Message --------
>>>>> From: Joan Touzet <wo...@apache.org>
>>>>> Sent: June 16, 2015 5:38:28 PM GMT+03:00
>>>>> To: user@couchdb.apache.org
>>>>> Subject: Re: CouchDB utc_id behavior
>>>>>
>>>>> now() is not guaranteed to be monotonically increasing if the system
>>>>> clock rolls backwards, which various things can cause.
>>>>>
>>>>> You should look into setting up ntpd for your two machines at the very
>>>>> least.
>>>>>
>>>>> -Joan
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Nick North" <no...@gmail.com>
>>>>>> To: user@couchdb.apache.org
>>>>>> Sent: Monday, June 15, 2015 12:14:50 PM
>>>>>> Subject: Re: CouchDB utc_id behavior
>>>>>>
>>>>>> The utc_id algorithm uses Erlang's now() function for generating
>>>>>> timestamps. This is guaranteed to be monotonic increasing, but not
>>>>>> necessarily to be in very close correspondence with the operating
>>>>>> system
>>>>>> clock all the time, especially if you call it very frequently.
>>>>>> However, I'm
>>>>>> surprised that calls seconds apart are giving this problem. Are your
>>>>>> machines VMs? There can be clock problems when they are suspended and
>>>>>> reactivated, with clocks initially having the time when the machine
>>>>>> was
>>>>>> suspended, and then jumping forward, but that's unlikely if they are
>>>>>> in
>>>>>> fairly constant use. For what it's worth, I use utc_id timestamps for
>>>>>> sorting documents, and have not seen this problem, but that doesn't
>>>>>> help
>>>>>> you very much.
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>> On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> nope, this is not the case.
>>>>>>> The newer document has older ID, this is the problem.
>>>>>>>
>>>>>>> 05188de92ef02f < 05188e0805067f
>>>>>>>
>>>>>>> But
>>>>>>> 05188de92ef02f
>>>>>>> was created after
>>>>>>> 05188e0805067f
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>>> *With best regards,*
>>>>>>> Kiril Stankov,
>>>>>>>
>>>>>>> On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
>>>>>>>> Time resolution is in microseconds, so difference in one second
>>>>>>>> generated notable "leap" forward.
>>>>>>>> --
>>>>>>>> ,,,^..^,,,
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov
>>>>>>>> <ki...@open-net.biz>
>>>>>>> wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I have two CouchDB (v1.6.1) servers, fully synchronized between
>>>>>>>>> them
>>>>>>>>> (master-to-master).
>>>>>>>>> The uuids algorithm is utc_id.
>>>>>>>>> The servers are synchronized via ntp and there is practically no
>>>>>>>>> time
>>>>>>> offset
>>>>>>>>> between them.
>>>>>>>>> I notice a strange behavior of the ID's of newly posted
>>>>>>>>> documents.
>>>>>>>>> In some cases, posting to server1, will generate ID, which is
>>>>>>>>> later
>>>>>>> than a
>>>>>>>>> subsequent post to server 2.
>>>>>>>>> E.g., posting to server 1 generates ID:
>>>>>>>>> 05188e0805067f_1
>>>>>>>>> and then, few seconds later, posting to server 2 generates:
>>>>>>>>> 05188de92ef02f_2
>>>>>>>>> As you see, the timestamp of the second message is earlier than
>>>>>>>>> the
>>>>>>> first
>>>>>>>>> (_1 & _2 are suffixes for the two servers).
>>>>>>>>> This is causing me a big mess, as I use the timestamp to sort
>>>>>>>>> and order
>>>>>>>>> documents.
>>>>>>>>> Any idea why this happens?
>>>>>>>>> Can someone, please, shed more light on how CouchDB "reads" the
>>>>>>>>> time
>>>>>>> for the
>>>>>>>>> generation of the ID?
>>>>>>>>> Or if you have an idea what may be causing this behavior.
>>>>>>>>>
>>>>>>>>> Thanks in advance!
>>>>>>>>>
>>>>>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>>>>> *With best regards,*
>>>>>>>>> Kiril Stankov,
>>>>>>>>>
>>>>>>>
>>
>>
>
Re: CouchDB utc_id behavior
Posted by Kiril Stankov <ki...@open-net.biz>.
Solved.
If Couch starts ~ 30 sec. after boot completion, the problem seems to go
away.
Thanks all.
------------------------------------------------------------------------
*With best regards,*
Kiril Stankov
On 18-Jun-15 2:15 PM, Kiril Stankov wrote:
> Yes, the one from 12:59 is Ok. The one from 12:51 is not.
> I will try now to delay the start of Couchdb with a minute after all
> other daemons started and see if it will help.
> May be the hardware clock is "off" and it takes time until ntpd syncs
> it and CouchDB starts in between.
>
>
> ------------------------------------------------------------------------
> *With best regards,*
> Kiril Stankov,
>
> On 17-Jun-15 6:41 PM, Adam Kocoloski wrote:
>> Do you happen to know if either one of these was correct-ish? Do you
>> see any timestamps in the access logs that are also “off”?
>>
>> 05188de92ef02f -> Mon, 15 Jun 2015 12:51:05 GMT
>> 05188e0805067f -> Mon, 15 Jun 2015 12:59:42 GMT
>>
>> Adam
>>
>>> On Jun 16, 2015, at 12:44 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>>>
>>> Ubuntu, couch 1.6.1, apt-get update few days ago.
>>> --
>>> Regards,
>>>
>>> Kiril Stankov,
>>> OpenNet Software ltd.
>>>
>>>
>>> -------- Original Message --------
>>> From: Nick North <no...@gmail.com>
>>> Sent: June 16, 2015 7:41:50 PM GMT+03:00
>>> To: user@couchdb.apache.org, Joan Touzet <wo...@apache.org>
>>> Subject: Re: CouchDB utc_id behavior
>>>
>>> Thanks for the correction Joan - I had forgotten about the
>>> possibility of
>>> the clock jumping backwards. However, in this case the obvious
>>> causes don't
>>> seem to apply, and I'm slightly at a loss. I can't hypothesise a
>>> mechanism
>>> that would cause now() to return these times if the OS clocks are in
>>> sync.
>>> Kiril - what setup are you running on?
>>>
>>> Nick
>>>
>>> On Tue, 16 Jun 2015 at 15:48 Kiril Stankov <ki...@open-net.biz> wrote:
>>>
>>>> Hi,
>>>> As I wrote, ntpd is running and both machines have synced time.
>>>> They were
>>>> not down for weeks.
>>>> --
>>>> Regards,
>>>>
>>>> Kiril Stankov,
>>>> OpenNet Software ltd.
>>>>
>>>>
>>>> -------- Original Message --------
>>>> From: Joan Touzet <wo...@apache.org>
>>>> Sent: June 16, 2015 5:38:28 PM GMT+03:00
>>>> To: user@couchdb.apache.org
>>>> Subject: Re: CouchDB utc_id behavior
>>>>
>>>> now() is not guaranteed to be monotonically increasing if the system
>>>> clock rolls backwards, which various things can cause.
>>>>
>>>> You should look into setting up ntpd for your two machines at the very
>>>> least.
>>>>
>>>> -Joan
>>>>
>>>> ----- Original Message -----
>>>>> From: "Nick North" <no...@gmail.com>
>>>>> To: user@couchdb.apache.org
>>>>> Sent: Monday, June 15, 2015 12:14:50 PM
>>>>> Subject: Re: CouchDB utc_id behavior
>>>>>
>>>>> The utc_id algorithm uses Erlang's now() function for generating
>>>>> timestamps. This is guaranteed to be monotonic increasing, but not
>>>>> necessarily to be in very close correspondence with the operating
>>>>> system
>>>>> clock all the time, especially if you call it very frequently.
>>>>> However, I'm
>>>>> surprised that calls seconds apart are giving this problem. Are your
>>>>> machines VMs? There can be clock problems when they are suspended and
>>>>> reactivated, with clocks initially having the time when the machine
>>>>> was
>>>>> suspended, and then jumping forward, but that's unlikely if they are
>>>>> in
>>>>> fairly constant use. For what it's worth, I use utc_id timestamps for
>>>>> sorting documents, and have not seen this problem, but that doesn't
>>>>> help
>>>>> you very much.
>>>>>
>>>>> Nick
>>>>>
>>>>> On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> nope, this is not the case.
>>>>>> The newer document has older ID, this is the problem.
>>>>>>
>>>>>> 05188de92ef02f < 05188e0805067f
>>>>>>
>>>>>> But
>>>>>> 05188de92ef02f
>>>>>> was created after
>>>>>> 05188e0805067f
>>>>>>
>>>>>>
>>>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>>> *With best regards,*
>>>>>> Kiril Stankov,
>>>>>>
>>>>>> On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
>>>>>>> Time resolution is in microseconds, so difference in one second
>>>>>>> generated notable "leap" forward.
>>>>>>> --
>>>>>>> ,,,^..^,,,
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov
>>>>>>> <ki...@open-net.biz>
>>>>>> wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I have two CouchDB (v1.6.1) servers, fully synchronized between
>>>>>>>> them
>>>>>>>> (master-to-master).
>>>>>>>> The uuids algorithm is utc_id.
>>>>>>>> The servers are synchronized via ntp and there is practically no
>>>>>>>> time
>>>>>> offset
>>>>>>>> between them.
>>>>>>>> I notice a strange behavior of the ID's of newly posted
>>>>>>>> documents.
>>>>>>>> In some cases, posting to server1, will generate ID, which is
>>>>>>>> later
>>>>>> than a
>>>>>>>> subsequent post to server 2.
>>>>>>>> E.g., posting to server 1 generates ID:
>>>>>>>> 05188e0805067f_1
>>>>>>>> and then, few seconds later, posting to server 2 generates:
>>>>>>>> 05188de92ef02f_2
>>>>>>>> As you see, the timestamp of the second message is earlier than
>>>>>>>> the
>>>>>> first
>>>>>>>> (_1 & _2 are suffixes for the two servers).
>>>>>>>> This is causing me a big mess, as I use the timestamp to sort
>>>>>>>> and order
>>>>>>>> documents.
>>>>>>>> Any idea why this happens?
>>>>>>>> Can someone, please, shed more light on how CouchDB "reads" the
>>>>>>>> time
>>>>>> for the
>>>>>>>> generation of the ID?
>>>>>>>> Or if you have an idea what may be causing this behavior.
>>>>>>>>
>>>>>>>> Thanks in advance!
>>>>>>>>
>>>>>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>>>>> *With best regards,*
>>>>>>>> Kiril Stankov,
>>>>>>>>
>>>>>>
>
>
Re: DB size question
Posted by Adam Kocoloski <ko...@apache.org>.
Thanks Sharath. For the record I did misspeak; the default value for checkpoint_after is 10x the buffer size, not 10 (i.e., its measured in bytes). Cheers,
Adam
> On Jun 19, 2015, at 10:03 AM, Kiril Stankov <ki...@open-net.biz> wrote:
>
> Thanks!
> I'll try the suggestions.
> ------------------------------------------------------------------------
> *With best regards,*
> Kiril Stankov,
>
> On 19-Jun-15 6:05 AM, Sharath wrote:
>> Hi Kiril,
>>
>> I came across this issue when I was using couchdb to store large documents.
>> Various members from this forumn helped me.
>>
>> You can find the conversation here:
>> http://qnalist.com/questions/5836043/couchdb-database-size
>>
>> The following setting helped reduce my database file size:
>>
>> checkpoint_after = 5242880000
>> doc_buffer_size = 524288000
>>
>>
>> I haven't had to revisit this setting. However, the drawback is the RAM
>> consumed (largish during compacting). I used to compact twice daily but now
>> its once weekly.
>>
>> My application mostly inserts to the database.
>>
>> regards,
>> Sharath
>>
>> On Fri, Jun 19, 2015 at 6:23 AM, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>> Yep, it’s normal. The wasted space is due to the purely copy-on-write
>>> nature of the btree indexes that the database maintains. Two main things
>>> you can do to reduce the overhead:
>>>
>>> * use the _bulk_docs endpoint
>>> * choose a long common prefix for the _ids of the documents in a given
>>> payload
>>>
>>> Yes, periodic compaction and cleanup is a good practice. Compaction only
>>> requires 1-2 extra file descriptors. It will use up to `doc_buffer_size`
>>> bytes to store docs in memory (default 512k), and will fsync after if fills
>>> the buffer `checkpoint_after` times (default 10). A larger buffer should
>>> result in a slightly faster compaction and a slightly more compact file.
>>> You probably don’t want to bother changing the checkpoint frequency. Cheers,
>>>
>>> Adam
>>>
>>>> On Jun 18, 2015, at 2:11 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm importing now a big number of documents in CouchDB.
>>>> The documents have only single revision. And they will stay with single
>>> rev in one of the DB's
>>>> I notice that the Db size grows significantly, then, after compact drops
>>> by 70%.
>>>> This process - import of single version documents will occur once a week.
>>>>
>>>> Why is so much space wasted? Is it something normal?
>>>>
>>>> Is it a good practice to run periodically compact and cleanup?
>>>>
>>>> Is there some DB size limit, after which the compact and cleanup may
>>> cause issue or have problems to run? E.g. file descriptors, memory. How
>>> should I configure checkpoint_after, doc_buffer_size?
>>>> Thanks in advance.
>>>>
>>>
>
Re: DB size question
Posted by Kiril Stankov <ki...@open-net.biz>.
Thanks!
I'll try the suggestions.
------------------------------------------------------------------------
*With best regards,*
Kiril Stankov,
On 19-Jun-15 6:05 AM, Sharath wrote:
> Hi Kiril,
>
> I came across this issue when I was using couchdb to store large documents.
> Various members from this forumn helped me.
>
> You can find the conversation here:
> http://qnalist.com/questions/5836043/couchdb-database-size
>
> The following setting helped reduce my database file size:
>
> checkpoint_after = 5242880000
> doc_buffer_size = 524288000
>
>
> I haven't had to revisit this setting. However, the drawback is the RAM
> consumed (largish during compacting). I used to compact twice daily but now
> its once weekly.
>
> My application mostly inserts to the database.
>
> regards,
> Sharath
>
> On Fri, Jun 19, 2015 at 6:23 AM, Adam Kocoloski <ko...@apache.org> wrote:
>
>> Yep, it’s normal. The wasted space is due to the purely copy-on-write
>> nature of the btree indexes that the database maintains. Two main things
>> you can do to reduce the overhead:
>>
>> * use the _bulk_docs endpoint
>> * choose a long common prefix for the _ids of the documents in a given
>> payload
>>
>> Yes, periodic compaction and cleanup is a good practice. Compaction only
>> requires 1-2 extra file descriptors. It will use up to `doc_buffer_size`
>> bytes to store docs in memory (default 512k), and will fsync after if fills
>> the buffer `checkpoint_after` times (default 10). A larger buffer should
>> result in a slightly faster compaction and a slightly more compact file.
>> You probably don’t want to bother changing the checkpoint frequency. Cheers,
>>
>> Adam
>>
>>> On Jun 18, 2015, at 2:11 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>>>
>>> Hi,
>>>
>>> I'm importing now a big number of documents in CouchDB.
>>> The documents have only single revision. And they will stay with single
>> rev in one of the DB's
>>> I notice that the Db size grows significantly, then, after compact drops
>> by 70%.
>>> This process - import of single version documents will occur once a week.
>>>
>>> Why is so much space wasted? Is it something normal?
>>>
>>> Is it a good practice to run periodically compact and cleanup?
>>>
>>> Is there some DB size limit, after which the compact and cleanup may
>> cause issue or have problems to run? E.g. file descriptors, memory. How
>> should I configure checkpoint_after, doc_buffer_size?
>>> Thanks in advance.
>>>
>>
Re: DB size question
Posted by Sharath <sh...@gmail.com>.
Hi Kiril,
I came across this issue when I was using couchdb to store large documents.
Various members from this forumn helped me.
You can find the conversation here:
http://qnalist.com/questions/5836043/couchdb-database-size
The following setting helped reduce my database file size:
checkpoint_after = 5242880000
doc_buffer_size = 524288000
I haven't had to revisit this setting. However, the drawback is the RAM
consumed (largish during compacting). I used to compact twice daily but now
its once weekly.
My application mostly inserts to the database.
regards,
Sharath
On Fri, Jun 19, 2015 at 6:23 AM, Adam Kocoloski <ko...@apache.org> wrote:
> Yep, it’s normal. The wasted space is due to the purely copy-on-write
> nature of the btree indexes that the database maintains. Two main things
> you can do to reduce the overhead:
>
> * use the _bulk_docs endpoint
> * choose a long common prefix for the _ids of the documents in a given
> payload
>
> Yes, periodic compaction and cleanup is a good practice. Compaction only
> requires 1-2 extra file descriptors. It will use up to `doc_buffer_size`
> bytes to store docs in memory (default 512k), and will fsync after if fills
> the buffer `checkpoint_after` times (default 10). A larger buffer should
> result in a slightly faster compaction and a slightly more compact file.
> You probably don’t want to bother changing the checkpoint frequency. Cheers,
>
> Adam
>
> > On Jun 18, 2015, at 2:11 PM, Kiril Stankov <ki...@open-net.biz> wrote:
> >
> > Hi,
> >
> > I'm importing now a big number of documents in CouchDB.
> > The documents have only single revision. And they will stay with single
> rev in one of the DB's
> > I notice that the Db size grows significantly, then, after compact drops
> by 70%.
> >
> > This process - import of single version documents will occur once a week.
> >
> > Why is so much space wasted? Is it something normal?
> >
> > Is it a good practice to run periodically compact and cleanup?
> >
> > Is there some DB size limit, after which the compact and cleanup may
> cause issue or have problems to run? E.g. file descriptors, memory. How
> should I configure checkpoint_after, doc_buffer_size?
> >
> > Thanks in advance.
> >
>
>
Re: DB size question
Posted by Adam Kocoloski <ko...@apache.org>.
Yep, it’s normal. The wasted space is due to the purely copy-on-write nature of the btree indexes that the database maintains. Two main things you can do to reduce the overhead:
* use the _bulk_docs endpoint
* choose a long common prefix for the _ids of the documents in a given payload
Yes, periodic compaction and cleanup is a good practice. Compaction only requires 1-2 extra file descriptors. It will use up to `doc_buffer_size` bytes to store docs in memory (default 512k), and will fsync after if fills the buffer `checkpoint_after` times (default 10). A larger buffer should result in a slightly faster compaction and a slightly more compact file. You probably don’t want to bother changing the checkpoint frequency. Cheers,
Adam
> On Jun 18, 2015, at 2:11 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>
> Hi,
>
> I'm importing now a big number of documents in CouchDB.
> The documents have only single revision. And they will stay with single rev in one of the DB's
> I notice that the Db size grows significantly, then, after compact drops by 70%.
>
> This process - import of single version documents will occur once a week.
>
> Why is so much space wasted? Is it something normal?
>
> Is it a good practice to run periodically compact and cleanup?
>
> Is there some DB size limit, after which the compact and cleanup may cause issue or have problems to run? E.g. file descriptors, memory. How should I configure checkpoint_after, doc_buffer_size?
>
> Thanks in advance.
>
DB size question
Posted by Kiril Stankov <ki...@open-net.biz>.
Hi,
I'm importing now a big number of documents in CouchDB.
The documents have only single revision. And they will stay with single
rev in one of the DB's
I notice that the Db size grows significantly, then, after compact drops
by 70%.
This process - import of single version documents will occur once a week.
Why is so much space wasted? Is it something normal?
Is it a good practice to run periodically compact and cleanup?
Is there some DB size limit, after which the compact and cleanup may
cause issue or have problems to run? E.g. file descriptors, memory. How
should I configure checkpoint_after, doc_buffer_size?
Thanks in advance.
Re: CouchDB utc_id behavior
Posted by Kiril Stankov <ki...@open-net.biz>.
Yes, the one from 12:59 is Ok. The one from 12:51 is not.
I will try now to delay the start of Couchdb with a minute after all
other daemons started and see if it will help.
May be the hardware clock is "off" and it takes time until ntpd syncs it
and CouchDB starts in between.
------------------------------------------------------------------------
*With best regards,*
Kiril Stankov,
On 17-Jun-15 6:41 PM, Adam Kocoloski wrote:
> Do you happen to know if either one of these was correct-ish? Do you see any timestamps in the access logs that are also “off”?
>
> 05188de92ef02f -> Mon, 15 Jun 2015 12:51:05 GMT
> 05188e0805067f -> Mon, 15 Jun 2015 12:59:42 GMT
>
> Adam
>
>> On Jun 16, 2015, at 12:44 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>>
>> Ubuntu, couch 1.6.1, apt-get update few days ago.
>> --
>> Regards,
>>
>> Kiril Stankov,
>> OpenNet Software ltd.
>>
>>
>> -------- Original Message --------
>> From: Nick North <no...@gmail.com>
>> Sent: June 16, 2015 7:41:50 PM GMT+03:00
>> To: user@couchdb.apache.org, Joan Touzet <wo...@apache.org>
>> Subject: Re: CouchDB utc_id behavior
>>
>> Thanks for the correction Joan - I had forgotten about the possibility of
>> the clock jumping backwards. However, in this case the obvious causes don't
>> seem to apply, and I'm slightly at a loss. I can't hypothesise a mechanism
>> that would cause now() to return these times if the OS clocks are in sync.
>> Kiril - what setup are you running on?
>>
>> Nick
>>
>> On Tue, 16 Jun 2015 at 15:48 Kiril Stankov <ki...@open-net.biz> wrote:
>>
>>> Hi,
>>> As I wrote, ntpd is running and both machines have synced time. They were
>>> not down for weeks.
>>> --
>>> Regards,
>>>
>>> Kiril Stankov,
>>> OpenNet Software ltd.
>>>
>>>
>>> -------- Original Message --------
>>> From: Joan Touzet <wo...@apache.org>
>>> Sent: June 16, 2015 5:38:28 PM GMT+03:00
>>> To: user@couchdb.apache.org
>>> Subject: Re: CouchDB utc_id behavior
>>>
>>> now() is not guaranteed to be monotonically increasing if the system
>>> clock rolls backwards, which various things can cause.
>>>
>>> You should look into setting up ntpd for your two machines at the very
>>> least.
>>>
>>> -Joan
>>>
>>> ----- Original Message -----
>>>> From: "Nick North" <no...@gmail.com>
>>>> To: user@couchdb.apache.org
>>>> Sent: Monday, June 15, 2015 12:14:50 PM
>>>> Subject: Re: CouchDB utc_id behavior
>>>>
>>>> The utc_id algorithm uses Erlang's now() function for generating
>>>> timestamps. This is guaranteed to be monotonic increasing, but not
>>>> necessarily to be in very close correspondence with the operating
>>>> system
>>>> clock all the time, especially if you call it very frequently.
>>>> However, I'm
>>>> surprised that calls seconds apart are giving this problem. Are your
>>>> machines VMs? There can be clock problems when they are suspended and
>>>> reactivated, with clocks initially having the time when the machine
>>>> was
>>>> suspended, and then jumping forward, but that's unlikely if they are
>>>> in
>>>> fairly constant use. For what it's worth, I use utc_id timestamps for
>>>> sorting documents, and have not seen this problem, but that doesn't
>>>> help
>>>> you very much.
>>>>
>>>> Nick
>>>>
>>>> On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> nope, this is not the case.
>>>>> The newer document has older ID, this is the problem.
>>>>>
>>>>> 05188de92ef02f < 05188e0805067f
>>>>>
>>>>> But
>>>>> 05188de92ef02f
>>>>> was created after
>>>>> 05188e0805067f
>>>>>
>>>>>
>>>>>
>>> ------------------------------------------------------------------------
>>>>> *With best regards,*
>>>>> Kiril Stankov,
>>>>>
>>>>> On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
>>>>>> Time resolution is in microseconds, so difference in one second
>>>>>> generated notable "leap" forward.
>>>>>> --
>>>>>> ,,,^..^,,,
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov
>>>>>> <ki...@open-net.biz>
>>>>> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I have two CouchDB (v1.6.1) servers, fully synchronized between
>>>>>>> them
>>>>>>> (master-to-master).
>>>>>>> The uuids algorithm is utc_id.
>>>>>>> The servers are synchronized via ntp and there is practically no
>>>>>>> time
>>>>> offset
>>>>>>> between them.
>>>>>>> I notice a strange behavior of the ID's of newly posted
>>>>>>> documents.
>>>>>>> In some cases, posting to server1, will generate ID, which is
>>>>>>> later
>>>>> than a
>>>>>>> subsequent post to server 2.
>>>>>>> E.g., posting to server 1 generates ID:
>>>>>>> 05188e0805067f_1
>>>>>>> and then, few seconds later, posting to server 2 generates:
>>>>>>> 05188de92ef02f_2
>>>>>>> As you see, the timestamp of the second message is earlier than
>>>>>>> the
>>>>> first
>>>>>>> (_1 & _2 are suffixes for the two servers).
>>>>>>> This is causing me a big mess, as I use the timestamp to sort
>>>>>>> and order
>>>>>>> documents.
>>>>>>> Any idea why this happens?
>>>>>>> Can someone, please, shed more light on how CouchDB "reads" the
>>>>>>> time
>>>>> for the
>>>>>>> generation of the ID?
>>>>>>> Or if you have an idea what may be causing this behavior.
>>>>>>>
>>>>>>> Thanks in advance!
>>>>>>>
>>>>>>>
>>> ------------------------------------------------------------------------
>>>>>>> *With best regards,*
>>>>>>> Kiril Stankov,
>>>>>>>
>>>>>
Re: CouchDB utc_id behavior
Posted by Adam Kocoloski <ko...@apache.org>.
Do you happen to know if either one of these was correct-ish? Do you see any timestamps in the access logs that are also “off”?
05188de92ef02f -> Mon, 15 Jun 2015 12:51:05 GMT
05188e0805067f -> Mon, 15 Jun 2015 12:59:42 GMT
Adam
> On Jun 16, 2015, at 12:44 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>
> Ubuntu, couch 1.6.1, apt-get update few days ago.
> --
> Regards,
>
> Kiril Stankov,
> OpenNet Software ltd.
>
>
> -------- Original Message --------
> From: Nick North <no...@gmail.com>
> Sent: June 16, 2015 7:41:50 PM GMT+03:00
> To: user@couchdb.apache.org, Joan Touzet <wo...@apache.org>
> Subject: Re: CouchDB utc_id behavior
>
> Thanks for the correction Joan - I had forgotten about the possibility of
> the clock jumping backwards. However, in this case the obvious causes don't
> seem to apply, and I'm slightly at a loss. I can't hypothesise a mechanism
> that would cause now() to return these times if the OS clocks are in sync.
> Kiril - what setup are you running on?
>
> Nick
>
> On Tue, 16 Jun 2015 at 15:48 Kiril Stankov <ki...@open-net.biz> wrote:
>
>> Hi,
>> As I wrote, ntpd is running and both machines have synced time. They were
>> not down for weeks.
>> --
>> Regards,
>>
>> Kiril Stankov,
>> OpenNet Software ltd.
>>
>>
>> -------- Original Message --------
>> From: Joan Touzet <wo...@apache.org>
>> Sent: June 16, 2015 5:38:28 PM GMT+03:00
>> To: user@couchdb.apache.org
>> Subject: Re: CouchDB utc_id behavior
>>
>> now() is not guaranteed to be monotonically increasing if the system
>> clock rolls backwards, which various things can cause.
>>
>> You should look into setting up ntpd for your two machines at the very
>> least.
>>
>> -Joan
>>
>> ----- Original Message -----
>>> From: "Nick North" <no...@gmail.com>
>>> To: user@couchdb.apache.org
>>> Sent: Monday, June 15, 2015 12:14:50 PM
>>> Subject: Re: CouchDB utc_id behavior
>>>
>>> The utc_id algorithm uses Erlang's now() function for generating
>>> timestamps. This is guaranteed to be monotonic increasing, but not
>>> necessarily to be in very close correspondence with the operating
>>> system
>>> clock all the time, especially if you call it very frequently.
>>> However, I'm
>>> surprised that calls seconds apart are giving this problem. Are your
>>> machines VMs? There can be clock problems when they are suspended and
>>> reactivated, with clocks initially having the time when the machine
>>> was
>>> suspended, and then jumping forward, but that's unlikely if they are
>>> in
>>> fairly constant use. For what it's worth, I use utc_id timestamps for
>>> sorting documents, and have not seen this problem, but that doesn't
>>> help
>>> you very much.
>>>
>>> Nick
>>>
>>> On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> nope, this is not the case.
>>>> The newer document has older ID, this is the problem.
>>>>
>>>> 05188de92ef02f < 05188e0805067f
>>>>
>>>> But
>>>> 05188de92ef02f
>>>> was created after
>>>> 05188e0805067f
>>>>
>>>>
>>>>
>> ------------------------------------------------------------------------
>>>> *With best regards,*
>>>> Kiril Stankov,
>>>>
>>>> On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
>>>>> Time resolution is in microseconds, so difference in one second
>>>>> generated notable "leap" forward.
>>>>> --
>>>>> ,,,^..^,,,
>>>>>
>>>>>
>>>>> On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov
>>>>> <ki...@open-net.biz>
>>>> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I have two CouchDB (v1.6.1) servers, fully synchronized between
>>>>>> them
>>>>>> (master-to-master).
>>>>>> The uuids algorithm is utc_id.
>>>>>> The servers are synchronized via ntp and there is practically no
>>>>>> time
>>>> offset
>>>>>> between them.
>>>>>> I notice a strange behavior of the ID's of newly posted
>>>>>> documents.
>>>>>> In some cases, posting to server1, will generate ID, which is
>>>>>> later
>>>> than a
>>>>>> subsequent post to server 2.
>>>>>> E.g., posting to server 1 generates ID:
>>>>>> 05188e0805067f_1
>>>>>> and then, few seconds later, posting to server 2 generates:
>>>>>> 05188de92ef02f_2
>>>>>> As you see, the timestamp of the second message is earlier than
>>>>>> the
>>>> first
>>>>>> (_1 & _2 are suffixes for the two servers).
>>>>>> This is causing me a big mess, as I use the timestamp to sort
>>>>>> and order
>>>>>> documents.
>>>>>> Any idea why this happens?
>>>>>> Can someone, please, shed more light on how CouchDB "reads" the
>>>>>> time
>>>> for the
>>>>>> generation of the ID?
>>>>>> Or if you have an idea what may be causing this behavior.
>>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>>
>> ------------------------------------------------------------------------
>>>>>> *With best regards,*
>>>>>> Kiril Stankov,
>>>>>>
>>>>
>>>>
>>>
>>
Re: CouchDB utc_id behavior
Posted by Kiril Stankov <ki...@open-net.biz>.
Ubuntu, couch 1.6.1, apt-get update few days ago.
--
Regards,
Kiril Stankov,
OpenNet Software ltd.
-------- Original Message --------
From: Nick North <no...@gmail.com>
Sent: June 16, 2015 7:41:50 PM GMT+03:00
To: user@couchdb.apache.org, Joan Touzet <wo...@apache.org>
Subject: Re: CouchDB utc_id behavior
Thanks for the correction Joan - I had forgotten about the possibility of
the clock jumping backwards. However, in this case the obvious causes don't
seem to apply, and I'm slightly at a loss. I can't hypothesise a mechanism
that would cause now() to return these times if the OS clocks are in sync.
Kiril - what setup are you running on?
Nick
On Tue, 16 Jun 2015 at 15:48 Kiril Stankov <ki...@open-net.biz> wrote:
> Hi,
> As I wrote, ntpd is running and both machines have synced time. They were
> not down for weeks.
> --
> Regards,
>
> Kiril Stankov,
> OpenNet Software ltd.
>
>
> -------- Original Message --------
> From: Joan Touzet <wo...@apache.org>
> Sent: June 16, 2015 5:38:28 PM GMT+03:00
> To: user@couchdb.apache.org
> Subject: Re: CouchDB utc_id behavior
>
> now() is not guaranteed to be monotonically increasing if the system
> clock rolls backwards, which various things can cause.
>
> You should look into setting up ntpd for your two machines at the very
> least.
>
> -Joan
>
> ----- Original Message -----
> > From: "Nick North" <no...@gmail.com>
> > To: user@couchdb.apache.org
> > Sent: Monday, June 15, 2015 12:14:50 PM
> > Subject: Re: CouchDB utc_id behavior
> >
> > The utc_id algorithm uses Erlang's now() function for generating
> > timestamps. This is guaranteed to be monotonic increasing, but not
> > necessarily to be in very close correspondence with the operating
> > system
> > clock all the time, especially if you call it very frequently.
> > However, I'm
> > surprised that calls seconds apart are giving this problem. Are your
> > machines VMs? There can be clock problems when they are suspended and
> > reactivated, with clocks initially having the time when the machine
> > was
> > suspended, and then jumping forward, but that's unlikely if they are
> > in
> > fairly constant use. For what it's worth, I use utc_id timestamps for
> > sorting documents, and have not seen this problem, but that doesn't
> > help
> > you very much.
> >
> > Nick
> >
> > On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz>
> > wrote:
> >
> > > Hi,
> > >
> > > nope, this is not the case.
> > > The newer document has older ID, this is the problem.
> > >
> > > 05188de92ef02f < 05188e0805067f
> > >
> > > But
> > > 05188de92ef02f
> > > was created after
> > > 05188e0805067f
> > >
> > >
> > >
> ------------------------------------------------------------------------
> > > *With best regards,*
> > > Kiril Stankov,
> > >
> > > On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
> > > > Time resolution is in microseconds, so difference in one second
> > > > generated notable "leap" forward.
> > > > --
> > > > ,,,^..^,,,
> > > >
> > > >
> > > > On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov
> > > > <ki...@open-net.biz>
> > > wrote:
> > > >> Hi all,
> > > >>
> > > >> I have two CouchDB (v1.6.1) servers, fully synchronized between
> > > >> them
> > > >> (master-to-master).
> > > >> The uuids algorithm is utc_id.
> > > >> The servers are synchronized via ntp and there is practically no
> > > >> time
> > > offset
> > > >> between them.
> > > >> I notice a strange behavior of the ID's of newly posted
> > > >> documents.
> > > >> In some cases, posting to server1, will generate ID, which is
> > > >> later
> > > than a
> > > >> subsequent post to server 2.
> > > >> E.g., posting to server 1 generates ID:
> > > >> 05188e0805067f_1
> > > >> and then, few seconds later, posting to server 2 generates:
> > > >> 05188de92ef02f_2
> > > >> As you see, the timestamp of the second message is earlier than
> > > >> the
> > > first
> > > >> (_1 & _2 are suffixes for the two servers).
> > > >> This is causing me a big mess, as I use the timestamp to sort
> > > >> and order
> > > >> documents.
> > > >> Any idea why this happens?
> > > >> Can someone, please, shed more light on how CouchDB "reads" the
> > > >> time
> > > for the
> > > >> generation of the ID?
> > > >> Or if you have an idea what may be causing this behavior.
> > > >>
> > > >> Thanks in advance!
> > > >>
> > > >>
> ------------------------------------------------------------------------
> > > >> *With best regards,*
> > > >> Kiril Stankov,
> > > >>
> > >
> > >
> >
>
Re: CouchDB utc_id behavior
Posted by Nick North <no...@gmail.com>.
Thanks for the correction Joan - I had forgotten about the possibility of
the clock jumping backwards. However, in this case the obvious causes don't
seem to apply, and I'm slightly at a loss. I can't hypothesise a mechanism
that would cause now() to return these times if the OS clocks are in sync.
Kiril - what setup are you running on?
Nick
On Tue, 16 Jun 2015 at 15:48 Kiril Stankov <ki...@open-net.biz> wrote:
> Hi,
> As I wrote, ntpd is running and both machines have synced time. They were
> not down for weeks.
> --
> Regards,
>
> Kiril Stankov,
> OpenNet Software ltd.
>
>
> -------- Original Message --------
> From: Joan Touzet <wo...@apache.org>
> Sent: June 16, 2015 5:38:28 PM GMT+03:00
> To: user@couchdb.apache.org
> Subject: Re: CouchDB utc_id behavior
>
> now() is not guaranteed to be monotonically increasing if the system
> clock rolls backwards, which various things can cause.
>
> You should look into setting up ntpd for your two machines at the very
> least.
>
> -Joan
>
> ----- Original Message -----
> > From: "Nick North" <no...@gmail.com>
> > To: user@couchdb.apache.org
> > Sent: Monday, June 15, 2015 12:14:50 PM
> > Subject: Re: CouchDB utc_id behavior
> >
> > The utc_id algorithm uses Erlang's now() function for generating
> > timestamps. This is guaranteed to be monotonic increasing, but not
> > necessarily to be in very close correspondence with the operating
> > system
> > clock all the time, especially if you call it very frequently.
> > However, I'm
> > surprised that calls seconds apart are giving this problem. Are your
> > machines VMs? There can be clock problems when they are suspended and
> > reactivated, with clocks initially having the time when the machine
> > was
> > suspended, and then jumping forward, but that's unlikely if they are
> > in
> > fairly constant use. For what it's worth, I use utc_id timestamps for
> > sorting documents, and have not seen this problem, but that doesn't
> > help
> > you very much.
> >
> > Nick
> >
> > On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz>
> > wrote:
> >
> > > Hi,
> > >
> > > nope, this is not the case.
> > > The newer document has older ID, this is the problem.
> > >
> > > 05188de92ef02f < 05188e0805067f
> > >
> > > But
> > > 05188de92ef02f
> > > was created after
> > > 05188e0805067f
> > >
> > >
> > >
> ------------------------------------------------------------------------
> > > *With best regards,*
> > > Kiril Stankov,
> > >
> > > On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
> > > > Time resolution is in microseconds, so difference in one second
> > > > generated notable "leap" forward.
> > > > --
> > > > ,,,^..^,,,
> > > >
> > > >
> > > > On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov
> > > > <ki...@open-net.biz>
> > > wrote:
> > > >> Hi all,
> > > >>
> > > >> I have two CouchDB (v1.6.1) servers, fully synchronized between
> > > >> them
> > > >> (master-to-master).
> > > >> The uuids algorithm is utc_id.
> > > >> The servers are synchronized via ntp and there is practically no
> > > >> time
> > > offset
> > > >> between them.
> > > >> I notice a strange behavior of the ID's of newly posted
> > > >> documents.
> > > >> In some cases, posting to server1, will generate ID, which is
> > > >> later
> > > than a
> > > >> subsequent post to server 2.
> > > >> E.g., posting to server 1 generates ID:
> > > >> 05188e0805067f_1
> > > >> and then, few seconds later, posting to server 2 generates:
> > > >> 05188de92ef02f_2
> > > >> As you see, the timestamp of the second message is earlier than
> > > >> the
> > > first
> > > >> (_1 & _2 are suffixes for the two servers).
> > > >> This is causing me a big mess, as I use the timestamp to sort
> > > >> and order
> > > >> documents.
> > > >> Any idea why this happens?
> > > >> Can someone, please, shed more light on how CouchDB "reads" the
> > > >> time
> > > for the
> > > >> generation of the ID?
> > > >> Or if you have an idea what may be causing this behavior.
> > > >>
> > > >> Thanks in advance!
> > > >>
> > > >>
> ------------------------------------------------------------------------
> > > >> *With best regards,*
> > > >> Kiril Stankov,
> > > >>
> > >
> > >
> >
>
Re: CouchDB utc_id behavior
Posted by Kiril Stankov <ki...@open-net.biz>.
Hi,
As I wrote, ntpd is running and both machines have synced time. They were not down for weeks.
--
Regards,
Kiril Stankov,
OpenNet Software ltd.
-------- Original Message --------
From: Joan Touzet <wo...@apache.org>
Sent: June 16, 2015 5:38:28 PM GMT+03:00
To: user@couchdb.apache.org
Subject: Re: CouchDB utc_id behavior
now() is not guaranteed to be monotonically increasing if the system
clock rolls backwards, which various things can cause.
You should look into setting up ntpd for your two machines at the very
least.
-Joan
----- Original Message -----
> From: "Nick North" <no...@gmail.com>
> To: user@couchdb.apache.org
> Sent: Monday, June 15, 2015 12:14:50 PM
> Subject: Re: CouchDB utc_id behavior
>
> The utc_id algorithm uses Erlang's now() function for generating
> timestamps. This is guaranteed to be monotonic increasing, but not
> necessarily to be in very close correspondence with the operating
> system
> clock all the time, especially if you call it very frequently.
> However, I'm
> surprised that calls seconds apart are giving this problem. Are your
> machines VMs? There can be clock problems when they are suspended and
> reactivated, with clocks initially having the time when the machine
> was
> suspended, and then jumping forward, but that's unlikely if they are
> in
> fairly constant use. For what it's worth, I use utc_id timestamps for
> sorting documents, and have not seen this problem, but that doesn't
> help
> you very much.
>
> Nick
>
> On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz>
> wrote:
>
> > Hi,
> >
> > nope, this is not the case.
> > The newer document has older ID, this is the problem.
> >
> > 05188de92ef02f < 05188e0805067f
> >
> > But
> > 05188de92ef02f
> > was created after
> > 05188e0805067f
> >
> >
> > ------------------------------------------------------------------------
> > *With best regards,*
> > Kiril Stankov,
> >
> > On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
> > > Time resolution is in microseconds, so difference in one second
> > > generated notable "leap" forward.
> > > --
> > > ,,,^..^,,,
> > >
> > >
> > > On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov
> > > <ki...@open-net.biz>
> > wrote:
> > >> Hi all,
> > >>
> > >> I have two CouchDB (v1.6.1) servers, fully synchronized between
> > >> them
> > >> (master-to-master).
> > >> The uuids algorithm is utc_id.
> > >> The servers are synchronized via ntp and there is practically no
> > >> time
> > offset
> > >> between them.
> > >> I notice a strange behavior of the ID's of newly posted
> > >> documents.
> > >> In some cases, posting to server1, will generate ID, which is
> > >> later
> > than a
> > >> subsequent post to server 2.
> > >> E.g., posting to server 1 generates ID:
> > >> 05188e0805067f_1
> > >> and then, few seconds later, posting to server 2 generates:
> > >> 05188de92ef02f_2
> > >> As you see, the timestamp of the second message is earlier than
> > >> the
> > first
> > >> (_1 & _2 are suffixes for the two servers).
> > >> This is causing me a big mess, as I use the timestamp to sort
> > >> and order
> > >> documents.
> > >> Any idea why this happens?
> > >> Can someone, please, shed more light on how CouchDB "reads" the
> > >> time
> > for the
> > >> generation of the ID?
> > >> Or if you have an idea what may be causing this behavior.
> > >>
> > >> Thanks in advance!
> > >>
> > >> ------------------------------------------------------------------------
> > >> *With best regards,*
> > >> Kiril Stankov,
> > >>
> >
> >
>
Re: CouchDB utc_id behavior
Posted by Joan Touzet <wo...@apache.org>.
now() is not guaranteed to be monotonically increasing if the system
clock rolls backwards, which various things can cause.
You should look into setting up ntpd for your two machines at the very
least.
-Joan
----- Original Message -----
> From: "Nick North" <no...@gmail.com>
> To: user@couchdb.apache.org
> Sent: Monday, June 15, 2015 12:14:50 PM
> Subject: Re: CouchDB utc_id behavior
>
> The utc_id algorithm uses Erlang's now() function for generating
> timestamps. This is guaranteed to be monotonic increasing, but not
> necessarily to be in very close correspondence with the operating
> system
> clock all the time, especially if you call it very frequently.
> However, I'm
> surprised that calls seconds apart are giving this problem. Are your
> machines VMs? There can be clock problems when they are suspended and
> reactivated, with clocks initially having the time when the machine
> was
> suspended, and then jumping forward, but that's unlikely if they are
> in
> fairly constant use. For what it's worth, I use utc_id timestamps for
> sorting documents, and have not seen this problem, but that doesn't
> help
> you very much.
>
> Nick
>
> On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz>
> wrote:
>
> > Hi,
> >
> > nope, this is not the case.
> > The newer document has older ID, this is the problem.
> >
> > 05188de92ef02f < 05188e0805067f
> >
> > But
> > 05188de92ef02f
> > was created after
> > 05188e0805067f
> >
> >
> > ------------------------------------------------------------------------
> > *With best regards,*
> > Kiril Stankov,
> >
> > On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
> > > Time resolution is in microseconds, so difference in one second
> > > generated notable "leap" forward.
> > > --
> > > ,,,^..^,,,
> > >
> > >
> > > On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov
> > > <ki...@open-net.biz>
> > wrote:
> > >> Hi all,
> > >>
> > >> I have two CouchDB (v1.6.1) servers, fully synchronized between
> > >> them
> > >> (master-to-master).
> > >> The uuids algorithm is utc_id.
> > >> The servers are synchronized via ntp and there is practically no
> > >> time
> > offset
> > >> between them.
> > >> I notice a strange behavior of the ID's of newly posted
> > >> documents.
> > >> In some cases, posting to server1, will generate ID, which is
> > >> later
> > than a
> > >> subsequent post to server 2.
> > >> E.g., posting to server 1 generates ID:
> > >> 05188e0805067f_1
> > >> and then, few seconds later, posting to server 2 generates:
> > >> 05188de92ef02f_2
> > >> As you see, the timestamp of the second message is earlier than
> > >> the
> > first
> > >> (_1 & _2 are suffixes for the two servers).
> > >> This is causing me a big mess, as I use the timestamp to sort
> > >> and order
> > >> documents.
> > >> Any idea why this happens?
> > >> Can someone, please, shed more light on how CouchDB "reads" the
> > >> time
> > for the
> > >> generation of the ID?
> > >> Or if you have an idea what may be causing this behavior.
> > >>
> > >> Thanks in advance!
> > >>
> > >> ------------------------------------------------------------------------
> > >> *With best regards,*
> > >> Kiril Stankov,
> > >>
> >
> >
>
Re: CouchDB utc_id behavior
Posted by Nick North <no...@gmail.com>.
The utc_id algorithm uses Erlang's now() function for generating
timestamps. This is guaranteed to be monotonic increasing, but not
necessarily to be in very close correspondence with the operating system
clock all the time, especially if you call it very frequently. However, I'm
surprised that calls seconds apart are giving this problem. Are your
machines VMs? There can be clock problems when they are suspended and
reactivated, with clocks initially having the time when the machine was
suspended, and then jumping forward, but that's unlikely if they are in
fairly constant use. For what it's worth, I use utc_id timestamps for
sorting documents, and have not seen this problem, but that doesn't help
you very much.
Nick
On Mon, 15 Jun 2015 at 16:42 Kiril Stankov <ki...@open-net.biz> wrote:
> Hi,
>
> nope, this is not the case.
> The newer document has older ID, this is the problem.
>
> 05188de92ef02f < 05188e0805067f
>
> But
> 05188de92ef02f
> was created after
> 05188e0805067f
>
>
> ------------------------------------------------------------------------
> *With best regards,*
> Kiril Stankov,
>
> On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
> > Time resolution is in microseconds, so difference in one second
> > generated notable "leap" forward.
> > --
> > ,,,^..^,,,
> >
> >
> > On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov <ki...@open-net.biz>
> wrote:
> >> Hi all,
> >>
> >> I have two CouchDB (v1.6.1) servers, fully synchronized between them
> >> (master-to-master).
> >> The uuids algorithm is utc_id.
> >> The servers are synchronized via ntp and there is practically no time
> offset
> >> between them.
> >> I notice a strange behavior of the ID's of newly posted documents.
> >> In some cases, posting to server1, will generate ID, which is later
> than a
> >> subsequent post to server 2.
> >> E.g., posting to server 1 generates ID:
> >> 05188e0805067f_1
> >> and then, few seconds later, posting to server 2 generates:
> >> 05188de92ef02f_2
> >> As you see, the timestamp of the second message is earlier than the
> first
> >> (_1 & _2 are suffixes for the two servers).
> >> This is causing me a big mess, as I use the timestamp to sort and order
> >> documents.
> >> Any idea why this happens?
> >> Can someone, please, shed more light on how CouchDB "reads" the time
> for the
> >> generation of the ID?
> >> Or if you have an idea what may be causing this behavior.
> >>
> >> Thanks in advance!
> >>
> >> ------------------------------------------------------------------------
> >> *With best regards,*
> >> Kiril Stankov,
> >>
>
>
Re: CouchDB utc_id behavior
Posted by Kiril Stankov <ki...@open-net.biz>.
Hi,
nope, this is not the case.
The newer document has older ID, this is the problem.
05188de92ef02f < 05188e0805067f
But
05188de92ef02f
was created after
05188e0805067f
------------------------------------------------------------------------
*With best regards,*
Kiril Stankov,
On 15-Jun-15 6:08 PM, Alexander Shorin wrote:
> Time resolution is in microseconds, so difference in one second
> generated notable "leap" forward.
> --
> ,,,^..^,,,
>
>
> On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov <ki...@open-net.biz> wrote:
>> Hi all,
>>
>> I have two CouchDB (v1.6.1) servers, fully synchronized between them
>> (master-to-master).
>> The uuids algorithm is utc_id.
>> The servers are synchronized via ntp and there is practically no time offset
>> between them.
>> I notice a strange behavior of the ID's of newly posted documents.
>> In some cases, posting to server1, will generate ID, which is later than a
>> subsequent post to server 2.
>> E.g., posting to server 1 generates ID:
>> 05188e0805067f_1
>> and then, few seconds later, posting to server 2 generates:
>> 05188de92ef02f_2
>> As you see, the timestamp of the second message is earlier than the first
>> (_1 & _2 are suffixes for the two servers).
>> This is causing me a big mess, as I use the timestamp to sort and order
>> documents.
>> Any idea why this happens?
>> Can someone, please, shed more light on how CouchDB "reads" the time for the
>> generation of the ID?
>> Or if you have an idea what may be causing this behavior.
>>
>> Thanks in advance!
>>
>> ------------------------------------------------------------------------
>> *With best regards,*
>> Kiril Stankov,
>>
Re: CouchDB utc_id behavior
Posted by Alexander Shorin <kx...@gmail.com>.
Time resolution is in microseconds, so difference in one second
generated notable "leap" forward.
--
,,,^..^,,,
On Mon, Jun 15, 2015 at 5:10 PM, Kiril Stankov <ki...@open-net.biz> wrote:
> Hi all,
>
> I have two CouchDB (v1.6.1) servers, fully synchronized between them
> (master-to-master).
> The uuids algorithm is utc_id.
> The servers are synchronized via ntp and there is practically no time offset
> between them.
> I notice a strange behavior of the ID's of newly posted documents.
> In some cases, posting to server1, will generate ID, which is later than a
> subsequent post to server 2.
> E.g., posting to server 1 generates ID:
> 05188e0805067f_1
> and then, few seconds later, posting to server 2 generates:
> 05188de92ef02f_2
> As you see, the timestamp of the second message is earlier than the first
> (_1 & _2 are suffixes for the two servers).
> This is causing me a big mess, as I use the timestamp to sort and order
> documents.
> Any idea why this happens?
> Can someone, please, shed more light on how CouchDB "reads" the time for the
> generation of the ID?
> Or if you have an idea what may be causing this behavior.
>
> Thanks in advance!
>
> ------------------------------------------------------------------------
> *With best regards,*
> Kiril Stankov,
>
CouchDB utc_id behavior
Posted by Kiril Stankov <ki...@open-net.biz>.
Hi all,
I have two CouchDB (v1.6.1) servers, fully synchronized between them
(master-to-master).
The uuids algorithm is utc_id.
The servers are synchronized via ntp and there is practically no time
offset between them.
I notice a strange behavior of the ID's of newly posted documents.
In some cases, posting to server1, will generate ID, which is later than
a subsequent post to server 2.
E.g., posting to server 1 generates ID:
05188e0805067f_1
and then, few seconds later, posting to server 2 generates:
05188de92ef02f_2
As you see, the timestamp of the second message is earlier than the
first (_1 & _2 are suffixes for the two servers).
This is causing me a big mess, as I use the timestamp to sort and order
documents.
Any idea why this happens?
Can someone, please, shed more light on how CouchDB "reads" the time for
the generation of the ID?
Or if you have an idea what may be causing this behavior.
Thanks in advance!
------------------------------------------------------------------------
*With best regards,*
Kiril Stankov,
Re: CouchDB questions
Posted by Aurélien Bénel <au...@utt.fr>.
Hi Ram,
A nice introductory example ("Joe the plumber") is given in the official documentation :
http://docs.couchdb.org/en/1.6.1/intro/why.html#a-better-fit-for-common-applications
I would also recommend you to look at the "banking recipe" in the (outdated) CouchDB book, it helped me a lot in understanding the difference between MapReduce approach (store as it comes and index/aggregate later) and the relational approach.
http://guide.couchdb.org/draft/recipes.html#banking
Regards,
Aurélien
Re: CouchDB questions
Posted by Harald Kisch <ha...@gmail.com>.
Hi Ram,
the mayor advantage using a NoSQL-Database is to be schema-less.
Any software product will have evolutionary changes.
If you want to have fun a long time with your project, you shuld'nt use a
schema which is always necessary in the RDBMS world.
You should also consider, that nearly all knowledge you have about the
RDMBS world is simply not necessary anymore and may be contraproductive in
the NoSQL world. So you need to truncate your 'R' from your RDBMS knowledge
and you will get a new database world where you will find that modifiction
of your data-structures will become very simple and you can relax at any
evulutionary change in your software.
Best,
Harald
On Sat, Jun 13, 2015 at 1:48 PM, Ram Rachum <ra...@rachum.com> wrote:
> Happy you're enjoying this Andrew :)
>
> To be honest, right now I'm not seeing the advantage of using CouchDB for
> my case. I'll have to do things so differently, and then I'll still have to
> implement a transaction model on my own to implement undo functionality. So
> I'm not really seeing the advantages of the non-relational model here.
>
> On Sat, Jun 13, 2015 at 2:24 PM, Andrew Myers <am...@gmail.com> wrote:
>
> > Just wanted to say that I'm quietly watching this conversation and
> learning
> > quite a lot. Very helpful. Thanks all!
> >
>
Re: CouchDB questions
Posted by Ram Rachum <ra...@rachum.com>.
Happy you're enjoying this Andrew :)
To be honest, right now I'm not seeing the advantage of using CouchDB for
my case. I'll have to do things so differently, and then I'll still have to
implement a transaction model on my own to implement undo functionality. So
I'm not really seeing the advantages of the non-relational model here.
On Sat, Jun 13, 2015 at 2:24 PM, Andrew Myers <am...@gmail.com> wrote:
> Just wanted to say that I'm quietly watching this conversation and learning
> quite a lot. Very helpful. Thanks all!
>
Re: CouchDB questions
Posted by Andrew Myers <am...@gmail.com>.
Just wanted to say that I'm quietly watching this conversation and learning
quite a lot. Very helpful. Thanks all!
Re: CouchDB questions
Posted by Ram Rachum <ra...@rachum.com>.
Hmm, I wouldn't use a foreign key for tags in Postgres either, just some
kind of list (json or I believe there's something else too.)
Thanks for your help,
Ram.
On Sat, Jun 13, 2015 at 2:13 PM, Johannes Jörg Schmidt <schmidt@netzmerk.com
> wrote:
> Hi Ram,
>
> translations and tags are good examples where I would not use a
> relationship in CouchDB in most cases but instead include them in an object
> or array instead.
> But maybe that's what you would do with PostgreSQL anyway, since it
> supports JSON natively?
> Am 13.06.2015 12:23 schrieb "Ram Rachum" <ra...@rachum.com>:
>
> Thanks Aurélien!
>
> Can you please give me an example of a case where you'd use a relationship
> in PostgreSQL, but wouldn't if you were using CouchDB? This might help me
> understand the approach. (I tried reading about it but couldn't
> understand.)
>
>
> Thanks,
> Ram.
>
>
> On Sat, Jun 13, 2015 at 12:39 PM, Aurélien Bénel <au...@utt.fr>
> wrote:
>
> > >>>
> > http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
> > >> Is this how I'm supposed to use CouchDB? Because isn't this
> relational?
> > I think that if I'm pushing CouchDB to be like PostgreSQL, then maybe I'm
> > doing it wrong and I should either use PostgreSQL idiomatically or
> CouchDB
> > idiomatically.
> > >
> > > The fact that CouchDB is a document based database does not mean you
> > cannot model relations with it when it makes sense.
> >
> > Ram, Johannes, I'm afraid there is a term confusion between "relations"
> > and "relationships".
> >
> > The "relation" of "relational databases" is an algebraic notion for
> > something you would probably call a "table" (see
> > https://en.wikipedia.org/wiki/Relational_algebra).
> >
> > Having no "relations" (aka tables) doesn't mean you don't have
> > "relationships" (even if you have probably less of them).
> >
> >
> > Regards,
> >
> > Aurélien
>
Re: CouchDB questions
Posted by Johannes Jörg Schmidt <sc...@netzmerk.com>.
Hi Ram,
translations and tags are good examples where I would not use a
relationship in CouchDB in most cases but instead include them in an object
or array instead.
But maybe that's what you would do with PostgreSQL anyway, since it
supports JSON natively?
Am 13.06.2015 12:23 schrieb "Ram Rachum" <ra...@rachum.com>:
Thanks Aurélien!
Can you please give me an example of a case where you'd use a relationship
in PostgreSQL, but wouldn't if you were using CouchDB? This might help me
understand the approach. (I tried reading about it but couldn't understand.)
Thanks,
Ram.
On Sat, Jun 13, 2015 at 12:39 PM, Aurélien Bénel <au...@utt.fr>
wrote:
> >>>
> http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
> >> Is this how I'm supposed to use CouchDB? Because isn't this relational?
> I think that if I'm pushing CouchDB to be like PostgreSQL, then maybe I'm
> doing it wrong and I should either use PostgreSQL idiomatically or CouchDB
> idiomatically.
> >
> > The fact that CouchDB is a document based database does not mean you
> cannot model relations with it when it makes sense.
>
> Ram, Johannes, I'm afraid there is a term confusion between "relations"
> and "relationships".
>
> The "relation" of "relational databases" is an algebraic notion for
> something you would probably call a "table" (see
> https://en.wikipedia.org/wiki/Relational_algebra).
>
> Having no "relations" (aka tables) doesn't mean you don't have
> "relationships" (even if you have probably less of them).
>
>
> Regards,
>
> Aurélien
Re: CouchDB questions
Posted by Ram Rachum <ra...@rachum.com>.
Thanks Aurélien!
Can you please give me an example of a case where you'd use a relationship
in PostgreSQL, but wouldn't if you were using CouchDB? This might help me
understand the approach. (I tried reading about it but couldn't understand.)
Thanks,
Ram.
On Sat, Jun 13, 2015 at 12:39 PM, Aurélien Bénel <au...@utt.fr>
wrote:
> >>>
> http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
> >> Is this how I'm supposed to use CouchDB? Because isn't this relational?
> I think that if I'm pushing CouchDB to be like PostgreSQL, then maybe I'm
> doing it wrong and I should either use PostgreSQL idiomatically or CouchDB
> idiomatically.
> >
> > The fact that CouchDB is a document based database does not mean you
> cannot model relations with it when it makes sense.
>
> Ram, Johannes, I'm afraid there is a term confusion between "relations"
> and "relationships".
>
> The "relation" of "relational databases" is an algebraic notion for
> something you would probably call a "table" (see
> https://en.wikipedia.org/wiki/Relational_algebra).
>
> Having no "relations" (aka tables) doesn't mean you don't have
> "relationships" (even if you have probably less of them).
>
>
> Regards,
>
> Aurélien
Re: CouchDB questions
Posted by Aurélien Bénel <au...@utt.fr>.
>>> http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
>> Is this how I'm supposed to use CouchDB? Because isn't this relational? I think that if I'm pushing CouchDB to be like PostgreSQL, then maybe I'm doing it wrong and I should either use PostgreSQL idiomatically or CouchDB idiomatically.
>
> The fact that CouchDB is a document based database does not mean you cannot model relations with it when it makes sense.
Ram, Johannes, I'm afraid there is a term confusion between "relations" and "relationships".
The "relation" of "relational databases" is an algebraic notion for something you would probably call a "table" (see https://en.wikipedia.org/wiki/Relational_algebra).
Having no "relations" (aka tables) doesn't mean you don't have "relationships" (even if you have probably less of them).
Regards,
Aurélien
Re: CouchDB questions
Posted by Johannes Jörg Schmidt <sc...@netzmerk.com>.
The fact that CouchDB is a document based database does not mean you cannot
model relations with it when it makes sense.
Am 13.06.2015 11:10 schrieb "Ram Rachum" <ra...@rachum.com>:
> I read this:
> http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
>
> Is this how I'm supposed to use CouchDB? Because isn't this relational? I
> think that if I'm pushing CouchDB to be like PostgreSQL, then maybe I'm
> doing it wrong and I should either use PostgreSQL idiomatically or CouchDB
> idiomatically.
>
> On Sat, Jun 13, 2015 at 11:37 AM, Johannes Jörg Schmidt <
> schmidt@netzmerk.com> wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> >
> > Please read the sections "One To N Relations" [1], "N To N Relations"
> > [2] and "Linked Documents" [3] from CouchDB Best Practices.
> >
> > tl;dr you can and should link documents.
> >
> > [1]:
> > http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
> > [2]:
> > http://docs.ehealthafrica.org/couchdb-best-practices/#n-to-n-relations
> > [3]:
> > http://docs.ehealthafrica.org/couchdb-best-practices/#linked-documents
> >
> > On 13.06.2015 10:13, Ram Rachum wrote:
> > > Question: If it's schemaless, what do you do for foreign keys?
> > > Whenever I design a project, there's tons of foreign key. You have
> > > a user, and then he has objects associated with him, and then they
> > > have objects associated with them, etc. What do you do when you're
> > > using CouchDB. Foreign keys aren't idiomatic to CouchDB, right?
> > >
> > > On Sat, Jun 13, 2015 at 6:07 AM, Joel Wallis <jo...@gmail.com>
> > > wrote:
> > >
> > >> Ram, there's nothing between Python and CouchDB. You can just
> > >> make a simple search in the Python community to discover
> > >> alternatives to interact with CouchDB! Also, CouchDB's interface
> > >> is REST, so you probably already knows how to use it.
> > >>
> > >> PostgreSQL is very different than CouchDB. PostgreSQL is a
> > >> relational database with a wonderful SQL environment, and CouchDB
> > >> is a NoSQL/schemaless document-based database. Replication and
> > >> scalability in PostgreSQL is not so simple, although with CouchDB
> > >> its master-master simplicity is a core feature
> > >> <http://docs.couchdb.org/en/1.6.1/replication/intro.html>.
> > >> There's a big gap between them.
> > >>
> > >> I believe you must choose your database technology based in your
> > >> project needs/reality, so take a look at these aspects and tell
> > >> yourself which technology would fit your project needs.
> > >>
> > >> 2015-06-12 13:48 GMT-03:00 Ram Rachum <ra...@rachum.com>:
> > >>
> > >>> To be honest, in my experience when you use something through
> > >>> an adapter you always have to deal with more problems. So I'd
> > >>> prefer to avoid using
> > >> a
> > >>> Python library that talks to a JavaScript library. Also it
> > >>> seems that
> > >> undo
> > >>> isn't core functionality for PouchDB, I saw a separate repo for
> > >>> it. I
> > >> would
> > >>> have preferred to have it as core functionality. I think I'll
> > >>> just stick with Postgres. Thanks for your help!
> > >>>
> > >>> On Fri, Jun 12, 2015 at 7:30 PM, Johannes Jörg Schmidt <
> > >>> schmidt@netzmerk.com
> > >>>> wrote:
> > >>>
> > > Hi Ram,
> > >
> > > there is Python-PouchDB, which brings PouchDB to the Python land.
> > > I did not tried it but it sounds interesting:
> > > https://pythonhosted.org/Python-PouchDB/
> > >
> > > Johannes
> > >
> > > On 12.06.2015 18:16, Ram Rachum wrote:
> > >>>>>> Hi Joel,
> > >>>>>>
> > >>>>>> Thanks for the tip. But if I use PouchDB does it mean
> > >>>>>> that I need to use JavaScript? I'm very experienced with
> > >>>>>> Python, and hardly experienced with JavaScript. This
> > >>>>>> application is going to have lots of parts and logic to
> > >>>>>> it besides this feature, and I'd hate to have to program
> > >>>>>> the whole thing in JavaScript just because of PouchDB.
> > >>>>>>
> > >>>>>>
> > >>>>>> Thanks, Ram.
> > >>>>>>
> > >>>>>> On Fri, Jun 12, 2015 at 6:45 PM, Joel Wallis
> > >>>>>> <jo...@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> PS: PouchDB is a client-side JavaScript library that
> > >>>>>>> also runs with Node.js. It creates a database on
> > >>>>>>> client-side (using IndexedDB, WebSQL, or other storage
> > >>>>>>> strategies like localStorage or SQLite by adding
> > >>>>>>> plugins to it) with the ability to sync data with
> > >>>>>>> CoudhDB - or a CouchDB-compatible version of PouchDB
> > >>>>>>> called pouchdb-server (a separated project).
> > >>>>>>>
> > >>>>>>> http://pouchdb.com/ :-)
> > >>>>>>>
> > >>>>>>> 2015-06-12 12:42 GMT-03:00 Joel Wallis
> > >>>>>>> <jo...@gmail.com>:
> > >>>>>>>
> > >>>>>>>> You need PouchDB. Write your app with NW.js and this
> > >>>>>>>> feature will be trivial to implement.
> > >>>>>>>>
> > >>>>>>>> 2015-06-12 8:02 GMT-03:00 Roald de Vries
> > >>>>>>>> <we...@gmail.com>:
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> On 12 Jun 2015, at 12:55, Ram Rachum
> > >>>>>>>>>> <ra...@rachum.com> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> So if I'll want undo functionality, I'll have to
> > >>>>>>>>>> create my own log
> > >>>>>>>>> separate
> > >>>>>>>>>> from CouchDB's native log?
> > >>>>>>>>>
> > >>>>>>>>> For example. I’m not sure if there are other
> > >>>>>>>>> standard solutions to this (pretty common)
> > >>>>>>>>> problem.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> -- Joel Wallis Jucá joelwallis.com
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> -- Joel Wallis Jucá joelwallis.com
> > >>>>>>>
> > >>>>>>
> > >
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> -- Joel Wallis Jucá joelwallis.com
> > >>
> > >
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2
> >
> > iQEcBAEBCAAGBQJVe+vmAAoJED+W7gN+c0gcOoYH/34mRHI6slCfVIucHaGZLNwc
> > tgDzFDsorcbSlskCY/pnlzYjJ5QYuwxkQbWzBPrz4ZhvcXktmn0odghwk3D/bsOF
> > s3H8JFegIJ5IM75o0cCbg0wmR3nQlO+g5qxANuNs3VFElMhJMuyUlhf/UoTK2bnj
> > RZIVcN/gW05IfVxYHHNoTYJ7t2306U4LafSGOg55vru3peZuJ8M/Lditdy1wNDCm
> > o2TgNj9qM4i2nOLNkYevWp4lhhiAhWVh9Pn1/8WkxbwuzWNFn78qA2xLvnd4an7S
> > 2Ia/Jnm0koOHkUBr7IzN0b3acV1bRfXIVjxGWv/aJseuLdZ6I3VFVh2C3IYoiQk=
> > =m+Z8
> > -----END PGP SIGNATURE-----
> >
>
Re: CouchDB questions
Posted by Ram Rachum <ra...@rachum.com>.
I read this:
http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
Is this how I'm supposed to use CouchDB? Because isn't this relational? I
think that if I'm pushing CouchDB to be like PostgreSQL, then maybe I'm
doing it wrong and I should either use PostgreSQL idiomatically or CouchDB
idiomatically.
On Sat, Jun 13, 2015 at 11:37 AM, Johannes Jörg Schmidt <
schmidt@netzmerk.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Please read the sections "One To N Relations" [1], "N To N Relations"
> [2] and "Linked Documents" [3] from CouchDB Best Practices.
>
> tl;dr you can and should link documents.
>
> [1]:
> http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
> [2]:
> http://docs.ehealthafrica.org/couchdb-best-practices/#n-to-n-relations
> [3]:
> http://docs.ehealthafrica.org/couchdb-best-practices/#linked-documents
>
> On 13.06.2015 10:13, Ram Rachum wrote:
> > Question: If it's schemaless, what do you do for foreign keys?
> > Whenever I design a project, there's tons of foreign key. You have
> > a user, and then he has objects associated with him, and then they
> > have objects associated with them, etc. What do you do when you're
> > using CouchDB. Foreign keys aren't idiomatic to CouchDB, right?
> >
> > On Sat, Jun 13, 2015 at 6:07 AM, Joel Wallis <jo...@gmail.com>
> > wrote:
> >
> >> Ram, there's nothing between Python and CouchDB. You can just
> >> make a simple search in the Python community to discover
> >> alternatives to interact with CouchDB! Also, CouchDB's interface
> >> is REST, so you probably already knows how to use it.
> >>
> >> PostgreSQL is very different than CouchDB. PostgreSQL is a
> >> relational database with a wonderful SQL environment, and CouchDB
> >> is a NoSQL/schemaless document-based database. Replication and
> >> scalability in PostgreSQL is not so simple, although with CouchDB
> >> its master-master simplicity is a core feature
> >> <http://docs.couchdb.org/en/1.6.1/replication/intro.html>.
> >> There's a big gap between them.
> >>
> >> I believe you must choose your database technology based in your
> >> project needs/reality, so take a look at these aspects and tell
> >> yourself which technology would fit your project needs.
> >>
> >> 2015-06-12 13:48 GMT-03:00 Ram Rachum <ra...@rachum.com>:
> >>
> >>> To be honest, in my experience when you use something through
> >>> an adapter you always have to deal with more problems. So I'd
> >>> prefer to avoid using
> >> a
> >>> Python library that talks to a JavaScript library. Also it
> >>> seems that
> >> undo
> >>> isn't core functionality for PouchDB, I saw a separate repo for
> >>> it. I
> >> would
> >>> have preferred to have it as core functionality. I think I'll
> >>> just stick with Postgres. Thanks for your help!
> >>>
> >>> On Fri, Jun 12, 2015 at 7:30 PM, Johannes Jörg Schmidt <
> >>> schmidt@netzmerk.com
> >>>> wrote:
> >>>
> > Hi Ram,
> >
> > there is Python-PouchDB, which brings PouchDB to the Python land.
> > I did not tried it but it sounds interesting:
> > https://pythonhosted.org/Python-PouchDB/
> >
> > Johannes
> >
> > On 12.06.2015 18:16, Ram Rachum wrote:
> >>>>>> Hi Joel,
> >>>>>>
> >>>>>> Thanks for the tip. But if I use PouchDB does it mean
> >>>>>> that I need to use JavaScript? I'm very experienced with
> >>>>>> Python, and hardly experienced with JavaScript. This
> >>>>>> application is going to have lots of parts and logic to
> >>>>>> it besides this feature, and I'd hate to have to program
> >>>>>> the whole thing in JavaScript just because of PouchDB.
> >>>>>>
> >>>>>>
> >>>>>> Thanks, Ram.
> >>>>>>
> >>>>>> On Fri, Jun 12, 2015 at 6:45 PM, Joel Wallis
> >>>>>> <jo...@gmail.com> wrote:
> >>>>>>
> >>>>>>> PS: PouchDB is a client-side JavaScript library that
> >>>>>>> also runs with Node.js. It creates a database on
> >>>>>>> client-side (using IndexedDB, WebSQL, or other storage
> >>>>>>> strategies like localStorage or SQLite by adding
> >>>>>>> plugins to it) with the ability to sync data with
> >>>>>>> CoudhDB - or a CouchDB-compatible version of PouchDB
> >>>>>>> called pouchdb-server (a separated project).
> >>>>>>>
> >>>>>>> http://pouchdb.com/ :-)
> >>>>>>>
> >>>>>>> 2015-06-12 12:42 GMT-03:00 Joel Wallis
> >>>>>>> <jo...@gmail.com>:
> >>>>>>>
> >>>>>>>> You need PouchDB. Write your app with NW.js and this
> >>>>>>>> feature will be trivial to implement.
> >>>>>>>>
> >>>>>>>> 2015-06-12 8:02 GMT-03:00 Roald de Vries
> >>>>>>>> <we...@gmail.com>:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 12 Jun 2015, at 12:55, Ram Rachum
> >>>>>>>>>> <ra...@rachum.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> So if I'll want undo functionality, I'll have to
> >>>>>>>>>> create my own log
> >>>>>>>>> separate
> >>>>>>>>>> from CouchDB's native log?
> >>>>>>>>>
> >>>>>>>>> For example. I’m not sure if there are other
> >>>>>>>>> standard solutions to this (pretty common)
> >>>>>>>>> problem.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -- Joel Wallis Jucá joelwallis.com
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -- Joel Wallis Jucá joelwallis.com
> >>>>>>>
> >>>>>>
> >
> >>>>
> >>>
> >>
> >>
> >>
> >> -- Joel Wallis Jucá joelwallis.com
> >>
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQEcBAEBCAAGBQJVe+vmAAoJED+W7gN+c0gcOoYH/34mRHI6slCfVIucHaGZLNwc
> tgDzFDsorcbSlskCY/pnlzYjJ5QYuwxkQbWzBPrz4ZhvcXktmn0odghwk3D/bsOF
> s3H8JFegIJ5IM75o0cCbg0wmR3nQlO+g5qxANuNs3VFElMhJMuyUlhf/UoTK2bnj
> RZIVcN/gW05IfVxYHHNoTYJ7t2306U4LafSGOg55vru3peZuJ8M/Lditdy1wNDCm
> o2TgNj9qM4i2nOLNkYevWp4lhhiAhWVh9Pn1/8WkxbwuzWNFn78qA2xLvnd4an7S
> 2Ia/Jnm0koOHkUBr7IzN0b3acV1bRfXIVjxGWv/aJseuLdZ6I3VFVh2C3IYoiQk=
> =m+Z8
> -----END PGP SIGNATURE-----
>
Re: CouchDB questions
Posted by Johannes Jörg Schmidt <sc...@netzmerk.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Please read the sections "One To N Relations" [1], "N To N Relations"
[2] and "Linked Documents" [3] from CouchDB Best Practices.
tl;dr you can and should link documents.
[1]:
http://docs.ehealthafrica.org/couchdb-best-practices/#one-to-n-relations
[2]:
http://docs.ehealthafrica.org/couchdb-best-practices/#n-to-n-relations
[3]:
http://docs.ehealthafrica.org/couchdb-best-practices/#linked-documents
On 13.06.2015 10:13, Ram Rachum wrote:
> Question: If it's schemaless, what do you do for foreign keys?
> Whenever I design a project, there's tons of foreign key. You have
> a user, and then he has objects associated with him, and then they
> have objects associated with them, etc. What do you do when you're
> using CouchDB. Foreign keys aren't idiomatic to CouchDB, right?
>
> On Sat, Jun 13, 2015 at 6:07 AM, Joel Wallis <jo...@gmail.com>
> wrote:
>
>> Ram, there's nothing between Python and CouchDB. You can just
>> make a simple search in the Python community to discover
>> alternatives to interact with CouchDB! Also, CouchDB's interface
>> is REST, so you probably already knows how to use it.
>>
>> PostgreSQL is very different than CouchDB. PostgreSQL is a
>> relational database with a wonderful SQL environment, and CouchDB
>> is a NoSQL/schemaless document-based database. Replication and
>> scalability in PostgreSQL is not so simple, although with CouchDB
>> its master-master simplicity is a core feature
>> <http://docs.couchdb.org/en/1.6.1/replication/intro.html>.
>> There's a big gap between them.
>>
>> I believe you must choose your database technology based in your
>> project needs/reality, so take a look at these aspects and tell
>> yourself which technology would fit your project needs.
>>
>> 2015-06-12 13:48 GMT-03:00 Ram Rachum <ra...@rachum.com>:
>>
>>> To be honest, in my experience when you use something through
>>> an adapter you always have to deal with more problems. So I'd
>>> prefer to avoid using
>> a
>>> Python library that talks to a JavaScript library. Also it
>>> seems that
>> undo
>>> isn't core functionality for PouchDB, I saw a separate repo for
>>> it. I
>> would
>>> have preferred to have it as core functionality. I think I'll
>>> just stick with Postgres. Thanks for your help!
>>>
>>> On Fri, Jun 12, 2015 at 7:30 PM, Johannes Jörg Schmidt <
>>> schmidt@netzmerk.com
>>>> wrote:
>>>
> Hi Ram,
>
> there is Python-PouchDB, which brings PouchDB to the Python land.
> I did not tried it but it sounds interesting:
> https://pythonhosted.org/Python-PouchDB/
>
> Johannes
>
> On 12.06.2015 18:16, Ram Rachum wrote:
>>>>>> Hi Joel,
>>>>>>
>>>>>> Thanks for the tip. But if I use PouchDB does it mean
>>>>>> that I need to use JavaScript? I'm very experienced with
>>>>>> Python, and hardly experienced with JavaScript. This
>>>>>> application is going to have lots of parts and logic to
>>>>>> it besides this feature, and I'd hate to have to program
>>>>>> the whole thing in JavaScript just because of PouchDB.
>>>>>>
>>>>>>
>>>>>> Thanks, Ram.
>>>>>>
>>>>>> On Fri, Jun 12, 2015 at 6:45 PM, Joel Wallis
>>>>>> <jo...@gmail.com> wrote:
>>>>>>
>>>>>>> PS: PouchDB is a client-side JavaScript library that
>>>>>>> also runs with Node.js. It creates a database on
>>>>>>> client-side (using IndexedDB, WebSQL, or other storage
>>>>>>> strategies like localStorage or SQLite by adding
>>>>>>> plugins to it) with the ability to sync data with
>>>>>>> CoudhDB - or a CouchDB-compatible version of PouchDB
>>>>>>> called pouchdb-server (a separated project).
>>>>>>>
>>>>>>> http://pouchdb.com/ :-)
>>>>>>>
>>>>>>> 2015-06-12 12:42 GMT-03:00 Joel Wallis
>>>>>>> <jo...@gmail.com>:
>>>>>>>
>>>>>>>> You need PouchDB. Write your app with NW.js and this
>>>>>>>> feature will be trivial to implement.
>>>>>>>>
>>>>>>>> 2015-06-12 8:02 GMT-03:00 Roald de Vries
>>>>>>>> <we...@gmail.com>:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 12 Jun 2015, at 12:55, Ram Rachum
>>>>>>>>>> <ra...@rachum.com> wrote:
>>>>>>>>>
>>>>>>>>>> So if I'll want undo functionality, I'll have to
>>>>>>>>>> create my own log
>>>>>>>>> separate
>>>>>>>>>> from CouchDB's native log?
>>>>>>>>>
>>>>>>>>> For example. I’m not sure if there are other
>>>>>>>>> standard solutions to this (pretty common)
>>>>>>>>> problem.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- Joel Wallis Jucá joelwallis.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Joel Wallis Jucá joelwallis.com
>>>>>>>
>>>>>>
>
>>>>
>>>
>>
>>
>>
>> -- Joel Wallis Jucá joelwallis.com
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJVe+vmAAoJED+W7gN+c0gcOoYH/34mRHI6slCfVIucHaGZLNwc
tgDzFDsorcbSlskCY/pnlzYjJ5QYuwxkQbWzBPrz4ZhvcXktmn0odghwk3D/bsOF
s3H8JFegIJ5IM75o0cCbg0wmR3nQlO+g5qxANuNs3VFElMhJMuyUlhf/UoTK2bnj
RZIVcN/gW05IfVxYHHNoTYJ7t2306U4LafSGOg55vru3peZuJ8M/Lditdy1wNDCm
o2TgNj9qM4i2nOLNkYevWp4lhhiAhWVh9Pn1/8WkxbwuzWNFn78qA2xLvnd4an7S
2Ia/Jnm0koOHkUBr7IzN0b3acV1bRfXIVjxGWv/aJseuLdZ6I3VFVh2C3IYoiQk=
=m+Z8
-----END PGP SIGNATURE-----
Re: CouchDB questions
Posted by Ram Rachum <ra...@rachum.com>.
Question: If it's schemaless, what do you do for foreign keys? Whenever I
design a project, there's tons of foreign key. You have a user, and then he
has objects associated with him, and then they have objects associated with
them, etc. What do you do when you're using CouchDB. Foreign keys aren't
idiomatic to CouchDB, right?
On Sat, Jun 13, 2015 at 6:07 AM, Joel Wallis <jo...@gmail.com> wrote:
> Ram, there's nothing between Python and CouchDB. You can just make a simple
> search in the Python community to discover alternatives to interact with
> CouchDB! Also, CouchDB's interface is REST, so you probably already knows
> how to use it.
>
> PostgreSQL is very different than CouchDB. PostgreSQL is a relational
> database with a wonderful SQL environment, and CouchDB is a
> NoSQL/schemaless document-based database. Replication and scalability in
> PostgreSQL is not so simple, although with CouchDB its master-master
> simplicity is a core feature
> <http://docs.couchdb.org/en/1.6.1/replication/intro.html>. There's a big
> gap between them.
>
> I believe you must choose your database technology based in your project
> needs/reality, so take a look at these aspects and tell yourself which
> technology would fit your project needs.
>
> 2015-06-12 13:48 GMT-03:00 Ram Rachum <ra...@rachum.com>:
>
> > To be honest, in my experience when you use something through an adapter
> > you always have to deal with more problems. So I'd prefer to avoid using
> a
> > Python library that talks to a JavaScript library. Also it seems that
> undo
> > isn't core functionality for PouchDB, I saw a separate repo for it. I
> would
> > have preferred to have it as core functionality. I think I'll just stick
> > with Postgres. Thanks for your help!
> >
> > On Fri, Jun 12, 2015 at 7:30 PM, Johannes Jörg Schmidt <
> > schmidt@netzmerk.com
> > > wrote:
> >
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA256
> > >
> > > Hi Ram,
> > >
> > > there is Python-PouchDB, which brings PouchDB to the Python land. I
> > > did not tried it but it sounds interesting:
> > > https://pythonhosted.org/Python-PouchDB/
> > >
> > > Johannes
> > >
> > > On 12.06.2015 18:16, Ram Rachum wrote:
> > > > Hi Joel,
> > > >
> > > > Thanks for the tip. But if I use PouchDB does it mean that I need
> > > > to use JavaScript? I'm very experienced with Python, and hardly
> > > > experienced with JavaScript. This application is going to have lots
> > > > of parts and logic to it besides this feature, and I'd hate to have
> > > > to program the whole thing in JavaScript just because of PouchDB.
> > > >
> > > >
> > > > Thanks, Ram.
> > > >
> > > > On Fri, Jun 12, 2015 at 6:45 PM, Joel Wallis <jo...@gmail.com>
> > > > wrote:
> > > >
> > > >> PS: PouchDB is a client-side JavaScript library that also runs
> > > >> with Node.js. It creates a database on client-side (using
> > > >> IndexedDB, WebSQL, or other storage strategies like localStorage
> > > >> or SQLite by adding plugins to it) with the ability to sync data
> > > >> with CoudhDB - or a CouchDB-compatible version of PouchDB called
> > > >> pouchdb-server (a separated project).
> > > >>
> > > >> http://pouchdb.com/ :-)
> > > >>
> > > >> 2015-06-12 12:42 GMT-03:00 Joel Wallis <jo...@gmail.com>:
> > > >>
> > > >>> You need PouchDB. Write your app with NW.js and this feature
> > > >>> will be trivial to implement.
> > > >>>
> > > >>> 2015-06-12 8:02 GMT-03:00 Roald de Vries
> > > >>> <we...@gmail.com>:
> > > >>>
> > > >>>>
> > > >>>>> On 12 Jun 2015, at 12:55, Ram Rachum <ra...@rachum.com>
> > > >>>>> wrote:
> > > >>>>
> > > >>>>> So if I'll want undo functionality, I'll have to create my
> > > >>>>> own log
> > > >>>> separate
> > > >>>>> from CouchDB's native log?
> > > >>>>
> > > >>>> For example. I’m not sure if there are other standard
> > > >>>> solutions to this (pretty common) problem.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> -- Joel Wallis Jucá joelwallis.com
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> -- Joel Wallis Jucá joelwallis.com
> > > >>
> > > >
> > >
> > > -----BEGIN PGP SIGNATURE-----
> > > Version: GnuPG v2
> > >
> > > iQEcBAEBCAAGBQJVewkIAAoJED+W7gN+c0gcb4YH/jU0kokung+ReE7ooDf7cPee
> > > cKBXeuM19QUHQvsFVLycsMiSw0xKgLx+cnVuYwR0ymZ0pEOIvt8AWxhLnQ4px/T4
> > > Wy53N7fJeLQ0txOJMNLUdiJWmRxMiGKltPvLtFYxeYZw0BWihVTPKP1IAO8fzLtO
> > > sUQZ+SIOZSNl3rD0xEcF22QpdACl0yaWt6S0GVxbaRwkKZzM884pdfOPgMD381CO
> > > Wyp7nzV64mfBTxe+STwCozHmaOvUgMx8NCsL+MZxX6Q4RHipcmYzrFUIkv4Bdl/0
> > > KaZEcnHwT55/8jQbVocNLfFWGjPFOGr+zWdf9hQlz9rpzzIc/yy8SrEZSMUuZWg=
> > > =yOmV
> > > -----END PGP SIGNATURE-----
> > >
> >
>
>
>
> --
> Joel Wallis Jucá
> joelwallis.com
>
Re: CouchDB questions
Posted by Joel Wallis <jo...@gmail.com>.
Ram, there's nothing between Python and CouchDB. You can just make a simple
search in the Python community to discover alternatives to interact with
CouchDB! Also, CouchDB's interface is REST, so you probably already knows
how to use it.
PostgreSQL is very different than CouchDB. PostgreSQL is a relational
database with a wonderful SQL environment, and CouchDB is a
NoSQL/schemaless document-based database. Replication and scalability in
PostgreSQL is not so simple, although with CouchDB its master-master
simplicity is a core feature
<http://docs.couchdb.org/en/1.6.1/replication/intro.html>. There's a big
gap between them.
I believe you must choose your database technology based in your project
needs/reality, so take a look at these aspects and tell yourself which
technology would fit your project needs.
2015-06-12 13:48 GMT-03:00 Ram Rachum <ra...@rachum.com>:
> To be honest, in my experience when you use something through an adapter
> you always have to deal with more problems. So I'd prefer to avoid using a
> Python library that talks to a JavaScript library. Also it seems that undo
> isn't core functionality for PouchDB, I saw a separate repo for it. I would
> have preferred to have it as core functionality. I think I'll just stick
> with Postgres. Thanks for your help!
>
> On Fri, Jun 12, 2015 at 7:30 PM, Johannes Jörg Schmidt <
> schmidt@netzmerk.com
> > wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> >
> > Hi Ram,
> >
> > there is Python-PouchDB, which brings PouchDB to the Python land. I
> > did not tried it but it sounds interesting:
> > https://pythonhosted.org/Python-PouchDB/
> >
> > Johannes
> >
> > On 12.06.2015 18:16, Ram Rachum wrote:
> > > Hi Joel,
> > >
> > > Thanks for the tip. But if I use PouchDB does it mean that I need
> > > to use JavaScript? I'm very experienced with Python, and hardly
> > > experienced with JavaScript. This application is going to have lots
> > > of parts and logic to it besides this feature, and I'd hate to have
> > > to program the whole thing in JavaScript just because of PouchDB.
> > >
> > >
> > > Thanks, Ram.
> > >
> > > On Fri, Jun 12, 2015 at 6:45 PM, Joel Wallis <jo...@gmail.com>
> > > wrote:
> > >
> > >> PS: PouchDB is a client-side JavaScript library that also runs
> > >> with Node.js. It creates a database on client-side (using
> > >> IndexedDB, WebSQL, or other storage strategies like localStorage
> > >> or SQLite by adding plugins to it) with the ability to sync data
> > >> with CoudhDB - or a CouchDB-compatible version of PouchDB called
> > >> pouchdb-server (a separated project).
> > >>
> > >> http://pouchdb.com/ :-)
> > >>
> > >> 2015-06-12 12:42 GMT-03:00 Joel Wallis <jo...@gmail.com>:
> > >>
> > >>> You need PouchDB. Write your app with NW.js and this feature
> > >>> will be trivial to implement.
> > >>>
> > >>> 2015-06-12 8:02 GMT-03:00 Roald de Vries
> > >>> <we...@gmail.com>:
> > >>>
> > >>>>
> > >>>>> On 12 Jun 2015, at 12:55, Ram Rachum <ra...@rachum.com>
> > >>>>> wrote:
> > >>>>
> > >>>>> So if I'll want undo functionality, I'll have to create my
> > >>>>> own log
> > >>>> separate
> > >>>>> from CouchDB's native log?
> > >>>>
> > >>>> For example. I’m not sure if there are other standard
> > >>>> solutions to this (pretty common) problem.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> -- Joel Wallis Jucá joelwallis.com
> > >>>
> > >>
> > >>
> > >>
> > >> -- Joel Wallis Jucá joelwallis.com
> > >>
> > >
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2
> >
> > iQEcBAEBCAAGBQJVewkIAAoJED+W7gN+c0gcb4YH/jU0kokung+ReE7ooDf7cPee
> > cKBXeuM19QUHQvsFVLycsMiSw0xKgLx+cnVuYwR0ymZ0pEOIvt8AWxhLnQ4px/T4
> > Wy53N7fJeLQ0txOJMNLUdiJWmRxMiGKltPvLtFYxeYZw0BWihVTPKP1IAO8fzLtO
> > sUQZ+SIOZSNl3rD0xEcF22QpdACl0yaWt6S0GVxbaRwkKZzM884pdfOPgMD381CO
> > Wyp7nzV64mfBTxe+STwCozHmaOvUgMx8NCsL+MZxX6Q4RHipcmYzrFUIkv4Bdl/0
> > KaZEcnHwT55/8jQbVocNLfFWGjPFOGr+zWdf9hQlz9rpzzIc/yy8SrEZSMUuZWg=
> > =yOmV
> > -----END PGP SIGNATURE-----
> >
>
--
Joel Wallis Jucá
joelwallis.com
Re: CouchDB questions
Posted by Ram Rachum <ra...@rachum.com>.
To be honest, in my experience when you use something through an adapter
you always have to deal with more problems. So I'd prefer to avoid using a
Python library that talks to a JavaScript library. Also it seems that undo
isn't core functionality for PouchDB, I saw a separate repo for it. I would
have preferred to have it as core functionality. I think I'll just stick
with Postgres. Thanks for your help!
On Fri, Jun 12, 2015 at 7:30 PM, Johannes Jörg Schmidt <schmidt@netzmerk.com
> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hi Ram,
>
> there is Python-PouchDB, which brings PouchDB to the Python land. I
> did not tried it but it sounds interesting:
> https://pythonhosted.org/Python-PouchDB/
>
> Johannes
>
> On 12.06.2015 18:16, Ram Rachum wrote:
> > Hi Joel,
> >
> > Thanks for the tip. But if I use PouchDB does it mean that I need
> > to use JavaScript? I'm very experienced with Python, and hardly
> > experienced with JavaScript. This application is going to have lots
> > of parts and logic to it besides this feature, and I'd hate to have
> > to program the whole thing in JavaScript just because of PouchDB.
> >
> >
> > Thanks, Ram.
> >
> > On Fri, Jun 12, 2015 at 6:45 PM, Joel Wallis <jo...@gmail.com>
> > wrote:
> >
> >> PS: PouchDB is a client-side JavaScript library that also runs
> >> with Node.js. It creates a database on client-side (using
> >> IndexedDB, WebSQL, or other storage strategies like localStorage
> >> or SQLite by adding plugins to it) with the ability to sync data
> >> with CoudhDB - or a CouchDB-compatible version of PouchDB called
> >> pouchdb-server (a separated project).
> >>
> >> http://pouchdb.com/ :-)
> >>
> >> 2015-06-12 12:42 GMT-03:00 Joel Wallis <jo...@gmail.com>:
> >>
> >>> You need PouchDB. Write your app with NW.js and this feature
> >>> will be trivial to implement.
> >>>
> >>> 2015-06-12 8:02 GMT-03:00 Roald de Vries
> >>> <we...@gmail.com>:
> >>>
> >>>>
> >>>>> On 12 Jun 2015, at 12:55, Ram Rachum <ra...@rachum.com>
> >>>>> wrote:
> >>>>
> >>>>> So if I'll want undo functionality, I'll have to create my
> >>>>> own log
> >>>> separate
> >>>>> from CouchDB's native log?
> >>>>
> >>>> For example. I’m not sure if there are other standard
> >>>> solutions to this (pretty common) problem.
> >>>
> >>>
> >>>
> >>>
> >>> -- Joel Wallis Jucá joelwallis.com
> >>>
> >>
> >>
> >>
> >> -- Joel Wallis Jucá joelwallis.com
> >>
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQEcBAEBCAAGBQJVewkIAAoJED+W7gN+c0gcb4YH/jU0kokung+ReE7ooDf7cPee
> cKBXeuM19QUHQvsFVLycsMiSw0xKgLx+cnVuYwR0ymZ0pEOIvt8AWxhLnQ4px/T4
> Wy53N7fJeLQ0txOJMNLUdiJWmRxMiGKltPvLtFYxeYZw0BWihVTPKP1IAO8fzLtO
> sUQZ+SIOZSNl3rD0xEcF22QpdACl0yaWt6S0GVxbaRwkKZzM884pdfOPgMD381CO
> Wyp7nzV64mfBTxe+STwCozHmaOvUgMx8NCsL+MZxX6Q4RHipcmYzrFUIkv4Bdl/0
> KaZEcnHwT55/8jQbVocNLfFWGjPFOGr+zWdf9hQlz9rpzzIc/yy8SrEZSMUuZWg=
> =yOmV
> -----END PGP SIGNATURE-----
>
Re: CouchDB questions
Posted by Johannes Jörg Schmidt <sc...@netzmerk.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Ram,
there is Python-PouchDB, which brings PouchDB to the Python land. I
did not tried it but it sounds interesting:
https://pythonhosted.org/Python-PouchDB/
Johannes
On 12.06.2015 18:16, Ram Rachum wrote:
> Hi Joel,
>
> Thanks for the tip. But if I use PouchDB does it mean that I need
> to use JavaScript? I'm very experienced with Python, and hardly
> experienced with JavaScript. This application is going to have lots
> of parts and logic to it besides this feature, and I'd hate to have
> to program the whole thing in JavaScript just because of PouchDB.
>
>
> Thanks, Ram.
>
> On Fri, Jun 12, 2015 at 6:45 PM, Joel Wallis <jo...@gmail.com>
> wrote:
>
>> PS: PouchDB is a client-side JavaScript library that also runs
>> with Node.js. It creates a database on client-side (using
>> IndexedDB, WebSQL, or other storage strategies like localStorage
>> or SQLite by adding plugins to it) with the ability to sync data
>> with CoudhDB - or a CouchDB-compatible version of PouchDB called
>> pouchdb-server (a separated project).
>>
>> http://pouchdb.com/ :-)
>>
>> 2015-06-12 12:42 GMT-03:00 Joel Wallis <jo...@gmail.com>:
>>
>>> You need PouchDB. Write your app with NW.js and this feature
>>> will be trivial to implement.
>>>
>>> 2015-06-12 8:02 GMT-03:00 Roald de Vries
>>> <we...@gmail.com>:
>>>
>>>>
>>>>> On 12 Jun 2015, at 12:55, Ram Rachum <ra...@rachum.com>
>>>>> wrote:
>>>>
>>>>> So if I'll want undo functionality, I'll have to create my
>>>>> own log
>>>> separate
>>>>> from CouchDB's native log?
>>>>
>>>> For example. I’m not sure if there are other standard
>>>> solutions to this (pretty common) problem.
>>>
>>>
>>>
>>>
>>> -- Joel Wallis Jucá joelwallis.com
>>>
>>
>>
>>
>> -- Joel Wallis Jucá joelwallis.com
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJVewkIAAoJED+W7gN+c0gcb4YH/jU0kokung+ReE7ooDf7cPee
cKBXeuM19QUHQvsFVLycsMiSw0xKgLx+cnVuYwR0ymZ0pEOIvt8AWxhLnQ4px/T4
Wy53N7fJeLQ0txOJMNLUdiJWmRxMiGKltPvLtFYxeYZw0BWihVTPKP1IAO8fzLtO
sUQZ+SIOZSNl3rD0xEcF22QpdACl0yaWt6S0GVxbaRwkKZzM884pdfOPgMD381CO
Wyp7nzV64mfBTxe+STwCozHmaOvUgMx8NCsL+MZxX6Q4RHipcmYzrFUIkv4Bdl/0
KaZEcnHwT55/8jQbVocNLfFWGjPFOGr+zWdf9hQlz9rpzzIc/yy8SrEZSMUuZWg=
=yOmV
-----END PGP SIGNATURE-----
Re: CouchDB questions
Posted by Ram Rachum <ra...@rachum.com>.
Hi Joel,
Thanks for the tip. But if I use PouchDB does it mean that I need to use
JavaScript? I'm very experienced with Python, and hardly experienced with
JavaScript. This application is going to have lots of parts and logic to it
besides this feature, and I'd hate to have to program the whole thing in
JavaScript just because of PouchDB.
Thanks,
Ram.
On Fri, Jun 12, 2015 at 6:45 PM, Joel Wallis <jo...@gmail.com> wrote:
> PS: PouchDB is a client-side JavaScript library that also runs with
> Node.js. It creates a database on client-side (using IndexedDB, WebSQL, or
> other storage strategies like localStorage or SQLite by adding plugins to
> it) with the ability to sync data with CoudhDB - or a CouchDB-compatible
> version of PouchDB called pouchdb-server (a separated project).
>
> http://pouchdb.com/ :-)
>
> 2015-06-12 12:42 GMT-03:00 Joel Wallis <jo...@gmail.com>:
>
> > You need PouchDB. Write your app with NW.js and this feature will be
> > trivial to implement.
> >
> > 2015-06-12 8:02 GMT-03:00 Roald de Vries <we...@gmail.com>:
> >
> >>
> >> > On 12 Jun 2015, at 12:55, Ram Rachum <ra...@rachum.com> wrote:
> >>
> >> > So if I'll want undo functionality, I'll have to create my own log
> >> separate
> >> > from CouchDB's native log?
> >>
> >> For example. I’m not sure if there are other standard solutions to this
> >> (pretty common) problem.
> >
> >
> >
> >
> > --
> > Joel Wallis Jucá
> > joelwallis.com
> >
>
>
>
> --
> Joel Wallis Jucá
> joelwallis.com
>
Re: CouchDB questions
Posted by Joel Wallis <jo...@gmail.com>.
PS: PouchDB is a client-side JavaScript library that also runs with
Node.js. It creates a database on client-side (using IndexedDB, WebSQL, or
other storage strategies like localStorage or SQLite by adding plugins to
it) with the ability to sync data with CoudhDB - or a CouchDB-compatible
version of PouchDB called pouchdb-server (a separated project).
http://pouchdb.com/ :-)
2015-06-12 12:42 GMT-03:00 Joel Wallis <jo...@gmail.com>:
> You need PouchDB. Write your app with NW.js and this feature will be
> trivial to implement.
>
> 2015-06-12 8:02 GMT-03:00 Roald de Vries <we...@gmail.com>:
>
>>
>> > On 12 Jun 2015, at 12:55, Ram Rachum <ra...@rachum.com> wrote:
>>
>> > So if I'll want undo functionality, I'll have to create my own log
>> separate
>> > from CouchDB's native log?
>>
>> For example. I’m not sure if there are other standard solutions to this
>> (pretty common) problem.
>
>
>
>
> --
> Joel Wallis Jucá
> joelwallis.com
>
--
Joel Wallis Jucá
joelwallis.com
Re: CouchDB questions
Posted by Joel Wallis <jo...@gmail.com>.
You need PouchDB. Write your app with NW.js and this feature will be
trivial to implement.
2015-06-12 8:02 GMT-03:00 Roald de Vries <we...@gmail.com>:
>
> > On 12 Jun 2015, at 12:55, Ram Rachum <ra...@rachum.com> wrote:
>
> > So if I'll want undo functionality, I'll have to create my own log
> separate
> > from CouchDB's native log?
>
> For example. I’m not sure if there are other standard solutions to this
> (pretty common) problem.
--
Joel Wallis Jucá
joelwallis.com
Re: CouchDB questions
Posted by Roald de Vries <we...@gmail.com>.
> On 12 Jun 2015, at 12:55, Ram Rachum <ra...@rachum.com> wrote:
> So if I'll want undo functionality, I'll have to create my own log separate
> from CouchDB's native log?
For example. I’m not sure if there are other standard solutions to this (pretty common) problem.
Re: CouchDB questions
Posted by Ram Rachum <ra...@rachum.com>.
Thanks Roald!
On Fri, Jun 12, 2015 at 1:43 PM, Roald de Vries <we...@gmail.com>
wrote:
> > 2. I'm assuming that CouchDB works by keeping a log of all transactions
> > made to the database. I want to provide undo functionality to the program
> > using this log of transactions. (i.e. every undo would look at the last
> > transaction and create a transaction that's the reverse of it.) This
> means
> > I need each transaction to save the old data, and I need the possibility
> to
> > access the transaction table. Is this possible?
>
> CouchDB uses an append-only file format, and a transaction replaces the
> previous version of a doc by a new one. The ‘transaction’ is not
> reversible, but if you’re lucky, the previous version of a document is
> still recoverable. You shouldn’t rely on this for undoing changes, though,
> because e.g. compaction will remove old versions, and replication won’t
> replicate old versions.
>
> Cheers, Roald
So if I'll want undo functionality, I'll have to create my own log separate
from CouchDB's native log?
Re: CouchDB questions
Posted by Roald de Vries <we...@gmail.com>.
Hi Ram,
> On 12 Jun 2015, at 12:15, Ram Rachum <ra...@rachum.com> wrote:
> I want the Django app to have a database,
You want Django to use CouchDB as its database? I think that's not supported officially, so that would probably mean not using the ORM and ModelForms.
> ... sync the two databases together.
>
> Is that something that CouchDB can handle?
Yes!
> If so, two questions:
>
> 1. What happens on merge fails? (i.e. when the two masters wrote data that
> contradicts each other.) I'd want to have a manual resolve, but what would
> be the interface to resolving it?
CouchDB doesn’t do merges, so you’ll have to resolve conflicts manually — which is good, because that’s what you want apparently ;-).
CouchDB uses MVCC. It keeps both/all of the conflicting versions of a document. You resolve the conflict by removing all but one of the conflicting versions, and possibly superseding winning one with a new version. See http://guide.couchdb.org/draft/conflicts.html
> 2. I'm assuming that CouchDB works by keeping a log of all transactions
> made to the database. I want to provide undo functionality to the program
> using this log of transactions. (i.e. every undo would look at the last
> transaction and create a transaction that's the reverse of it.) This means
> I need each transaction to save the old data, and I need the possibility to
> access the transaction table. Is this possible?
CouchDB uses an append-only file format, and a transaction replaces the previous version of a doc by a new one. The ‘transaction’ is not reversible, but if you’re lucky, the previous version of a document is still recoverable. You shouldn’t rely on this for undoing changes, though, because e.g. compaction will remove old versions, and replication won’t replicate old versions.
Cheers, Roald