Using CouchDB for transferring user logs via the internet

[Vladimir Kuznetsov <vo...@gmail.com>]

Hi guys, 

I'm new to couchdb. I'm planning to use CouchDB 2.0 to transfer some usage logs from a number of devices based on customer premises to the cloud solution for analysis. By logs I mean a constant stream of small JSON documents(up to 1K), I expect up to 100K of such documents from each device daily. 

1. I'm planning to implement this by setting up one way replication from couchdb on the device to the couchdb cluster(up to 8 instances) in the cloud.
2. as logs are being replicated on the cloud side I'm going to handle the data stream by subscribing to the _changes feed and passing device logs to the pipeline for the further processing. 
3. Once new log entries are taken and sent to the pipeline I basically don't need them in couchdb so I'm going to have data expiration by maintaining index on ctime and having separate job to do bulk deletion of documents older than a few days.

Now, I know in couchdb documents are not being really deleted, just marked as 'deleted' so the database will permanently grow. I've and option either to use periodic _purge(which I heard may be not safe especially in clustered environment) or implement this as monthly rotating database(which is more complex and I don't really want to follow this route).

My questions are:

- Is this a valid use case for CouchDB? I want to use it primarily because of its good replication capabilities, especially in not reliable environments with some periods of being offline etc. Otherwise I'll have to write the whole set of data sync APIs with buffering, retries etc myself.
- I find counter opinions in the internet - some people say CouhDB is a perfect tool for replication over the internet, other say - do use replication ONLY in the local network. What is the truth? Should I use CouchDB as a tool to sync data over not reliable network?
- Is this recommended practice to set up a chain of replication? Due to security considerations I want customer devices to replicate each to its own database in the cloud. Then I want those databases to replicate to the single central log database I'd subscribe for _changes. The reason is that it's easier for me to have a single source of _changes feed rather than multiple databases.
- Is using _purge safe in my case? From the official doc I read "In clustered or replicated environments it is very difficult to guarantee that a particular purged document has been removed from all replicas". I don't think this is a problem for me as I primarily care about database size so it shouldn't be critical if some documents fail to delete.
- Considering I may have up to 10G of new data daily(or ~10 million of new entries daily with 100 devices estimate) I'm probably going to expire documents older than 48 hours. How often do I need to run _purge?
- Does _purge block database for read/write while running? Does _compact (that I have to run after _purge) block?

thanks,
--Vovan

Re: Using CouchDB for transferring user logs via the internet

[Martin Broerse <ma...@gmail.com>]

We use pouch sync (ember-pouch) to couch in a bad sometimes offline wifi
situation and it is very reliable. Not sure about your 10G but it should be
OK if you have the bandwidth when you are online. We had to implement
indicators for users to let them know they are walking outside the wifi
connection. Indicator example https://bloggr.exmer.com/ Perhaps if you
setup a 10G couch to couch you need to implement indicators on the client
side too? Have not needed purge much so can't help you answer that.

- Martin

On Tue, Jun 20, 2017 at 9:24 PM, Vovan Vovan <vo...@gmail.com> wrote:

> Hi guys,
>
> I'm new to couchdb. I'm planning to use CouchDB 2.0 to transfer some usage
> logs from a number of devices based on customer premises to the cloud
> solution for analysis. By logs I mean a constant stream of small JSON
> documents(up to 1K), I expect up to 100K of such documents from each device
> daily.
>
> 1. I'm planning to implement this by setting up one way replication from
> couchdb on the device to the couchdb cluster(up to 8 instances) in the
> cloud.
> 2. as logs are being replicated on the cloud side I'm going to handle the
> data stream by subscribing to the _changes feed and passing device logs to
> the pipeline for the further processing.
> 3. Once new log entries are taken and sent to the pipeline I basically
> don't need them in couchdb so I'm going to have data expiration by
> maintaining index on ctime and having separate job to do bulk deletion of
> documents older than a few days.
>
> Now, I know in couchdb documents are not being really deleted, just marked
> as 'deleted' so the database will permanently grow. I've and option either
> to use periodic _purge(which I heard may be not safe especially in
> clustered environment) or implement this as monthly rotating database(which
> is more complex and I don't really want to follow this route).
>
> My questions are:
>
> - Is this a valid use case for CouchDB? I want to use it primarily because
> of its good replication capabilities, especially in not reliable
> environments with some periods of being offline etc. Otherwise I'll have to
> write the whole set of data sync APIs with buffering, retries etc myself.
> - I find counter opinions in the internet - some people say CouhDB is a
> perfect tool for replication over the internet, other say - do use
> replication ONLY in the local network. What is the truth? Should I use
> CouchDB as a tool to sync data over not reliable network?
> - Is this recommended practice to set up a chain of replication? Due to
> security considerations I want customer devices to replicate each to its
> own database in the cloud. Then I want those databases to replicate to the
> single central log database I'd subscribe for _changes. The reason is that
> it's easier for me to have a single source of _changes feed rather than
> multiple databases.
> - Is using _purge safe in my case? From the official doc I read "In
> clustered or replicated environments it is very difficult to guarantee that
> a particular purged document has been removed from all replicas". I don't
> think this is a problem for me as I primarily care about database size so
> it shouldn't be critical if some documents fail to delete.
> - Considering I may have up to 10G of new data daily(or ~10 million of new
> entries daily with 100 devices estimate) I'm probably going to expire
> documents older than 48 hours. How often do I need to run _purge?
> - Does _purge block database for read/write while running? Does _compact
> (that I have to run after _purge) block?
>
> thanks,
> --Vovan
>

Using CouchDB for transferring user logs via the internet

[Vovan Vovan <vo...@gmail.com>]

Hi guys,

I'm new to couchdb. I'm planning to use CouchDB 2.0 to transfer some usage
logs from a number of devices based on customer premises to the cloud
solution for analysis. By logs I mean a constant stream of small JSON
documents(up to 1K), I expect up to 100K of such documents from each device
daily.

1. I'm planning to implement this by setting up one way replication from
couchdb on the device to the couchdb cluster(up to 8 instances) in the
cloud.
2. as logs are being replicated on the cloud side I'm going to handle the
data stream by subscribing to the _changes feed and passing device logs to
the pipeline for the further processing.
3. Once new log entries are taken and sent to the pipeline I basically
don't need them in couchdb so I'm going to have data expiration by
maintaining index on ctime and having separate job to do bulk deletion of
documents older than a few days.

Now, I know in couchdb documents are not being really deleted, just marked
as 'deleted' so the database will permanently grow. I've and option either
to use periodic _purge(which I heard may be not safe especially in
clustered environment) or implement this as monthly rotating database(which
is more complex and I don't really want to follow this route).

My questions are:

- Is this a valid use case for CouchDB? I want to use it primarily because
of its good replication capabilities, especially in not reliable
environments with some periods of being offline etc. Otherwise I'll have to
write the whole set of data sync APIs with buffering, retries etc myself.
- I find counter opinions in the internet - some people say CouhDB is a
perfect tool for replication over the internet, other say - do use
replication ONLY in the local network. What is the truth? Should I use
CouchDB as a tool to sync data over not reliable network?
- Is this recommended practice to set up a chain of replication? Due to
security considerations I want customer devices to replicate each to its
own database in the cloud. Then I want those databases to replicate to the
single central log database I'd subscribe for _changes. The reason is that
it's easier for me to have a single source of _changes feed rather than
multiple databases.
- Is using _purge safe in my case? From the official doc I read "In
clustered or replicated environments it is very difficult to guarantee that
a particular purged document has been removed from all replicas". I don't
think this is a problem for me as I primarily care about database size so
it shouldn't be critical if some documents fail to delete.
- Considering I may have up to 10G of new data daily(or ~10 million of new
entries daily with 100 devices estimate) I'm probably going to expire
documents older than 48 hours. How often do I need to run _purge?
- Does _purge block database for read/write while running? Does _compact
(that I have to run after _purge) block?

thanks,
--Vovan

Re: Using CouchDB for transferring user logs via the internet

[Vladimir Kuznetsov <vo...@gmail.com>]

Hi Stefan

Thanks for pointing to the /_db_updates feature, I wasn't aware about it.

Yes, my only need for central db is to simplify polling for changes. No central views etc.  I'll see if I can efficiently poll on multiple databases changes by listening on /_db_updates feed:

 on each new entry where type  is 'updated' and db_name starts with 'some common prefix for dbs I want to track':
subscribe to the _changes feed of that db_name
stop listening for _changes after some period of inactivity

If this works I think rotating databases are not a big deal as I actually don't care what databases to listen, just pick any I see update notifications from. 

Thanks one more time.
--Vovan

P.S. after a little bit more reading I see that _purge may not be the right tool. Also I was a little bit confused thinking it's being invoked the same way _compact does i.e. once for entire database. This is not the case as I have to pass document id with all the all the revisions as a parameter, handle conflicts etc which makes things more complicated.

thanks,
--Vovan


> On Jun 20, 2017, at 2:12 PM, Stefan Klein <st...@gmail.com> wrote:
> 
> Hi,
> 
> commenting on individual topics inline, though i only have experience with
> CouchDB 1.6, maybe CouchDB 2.0 behaves a bit different.
> 
> 2017-06-20 20:34 GMT+02:00 Vladimir Kuznetsov <vo...@gmail.com>:
> 
>> 
>> Now, I know in couchdb documents are not being really deleted, just marked
>> as 'deleted' so the database will permanently grow. I've and option either
>> to use periodic _purge(which I heard may be not safe especially in
>> clustered environment) or implement this as monthly rotating database(which
>> is more complex and I don't really want to follow this route).
>> 
> 
> I think rotating databases are not that more complex, but see below.
> 
> 
>> My questions are:
>> 
>> - Is this a valid use case for CouchDB? I want to use it primarily because
>> of its good replication capabilities, especially in not reliable
>> environments with some periods of being offline etc. Otherwise I'll have to
>> write the whole set of data sync APIs with buffering, retries etc myself.
>> 
> 
> In our case, mobiles replicating to and from CouchDB, replication has
> proven to be very very reliable, it just works.
> If i had to implement that on my own it would have been much much worse.
> 
> 
>> - Is this recommended practice to set up a chain of replication? Due to
>> security considerations I want customer devices to replicate each to its
>> own database in the cloud. Then I want those databases to replicate to the
>> single central log database I'd subscribe for _changes. The reason is that
>> it's easier for me to have a single source of _changes feed rather than
>> multiple databases.
>> 
> 
> We do this, each user got his own database which we consider "outside", we
> monitor these databases for changes and take appropriate actions. :)
> On our server continuous replications from thousands of customer databases
> to a central database occupied to many connections¹ and the performance
> over all degraded, even if only "some" users where actually active. We now
> listen to DB changes (_db_updates), start replications for the DB in
> question and stop them again after a certain timeout, activity obviously
> resets the timeout.
> 
> If you consider the single central log database only so you have a single
> changes feed (you don't need centralized views etc.) I would skip the
> central database and just process _all_docs (or a view only containing
> unprocessed logentries) of any database an "updated" event was triggered on.
> 
> If you go that route, you are half way through rotating databases already,
> your backend doesn't care anymore on which database a change is triggered
> on.
> 
> 
>> - Is using _purge safe in my case? From the official doc I read "In
>> clustered or replicated environments it is very difficult to guarantee that
>> a particular purged document has been removed from all replicas". I don't
>> think this is a problem for me as I primarily care about database size so
>> it shouldn't be critical if some documents fail to delete.
>> 
> 
> _purge is the wrong tool for this job.
> From my understanding it's there as a last resort to get sensitive data out
> of a DB.
> 
> Regards,
> Stefan
> 
> [1] I think the main reason for this was actually the operating system, but
> it was faster, easier and more future proof to implement the described
> solution than to tune the OS to handle the connection, at least for me.

Re: Using CouchDB for transferring user logs via the internet

[Vladimir Kuznetsov <vo...@gmail.com>]

Hi Carlos,

One of the requirements for the system is to be able to work completely offline and sync when there's an internet connectivity. Right, this is a task for a queue, I would use one if I could find any implementation addressing offline replication use case.

thanks,
--Vovan

> On Jun 21, 2017, at 1:33 AM, Carlos Alonso <ca...@cabify.com> wrote:
> 
> Hi Vladimir.
> 
> I suppose you already have evaluated this option, but just in case... To me
> this sounds like a task for a queue. Have you thought about having a queue
> as your central logs storage (instead of the couchdb cluster on cloud you
> described)? To me it feels a more natural use case. From your devices you
> publish new messages (log entries) into the queue and then from your
> application you consume those messages to process them. The queue will
> already delete the processed messages for you, so you won't need to take
> care of it (no rotating dbs, purging, ...). There are many different tools
> out there so I guess you'll find one that provides your required
> durability/ordering requirements out of the box....
> 
> Hope it helps
> 
> On Tue, Jun 20, 2017 at 11:12 PM Stefan Klein <st...@gmail.com> wrote:
> 
>> Hi,
>> 
>> commenting on individual topics inline, though i only have experience with
>> CouchDB 1.6, maybe CouchDB 2.0 behaves a bit different.
>> 
>> 2017-06-20 20:34 GMT+02:00 Vladimir Kuznetsov <vo...@gmail.com>:
>> 
>>> 
>>> Now, I know in couchdb documents are not being really deleted, just
>> marked
>>> as 'deleted' so the database will permanently grow. I've and option
>> either
>>> to use periodic _purge(which I heard may be not safe especially in
>>> clustered environment) or implement this as monthly rotating
>> database(which
>>> is more complex and I don't really want to follow this route).
>>> 
>> 
>> I think rotating databases are not that more complex, but see below.
>> 
>> 
>>> My questions are:
>>> 
>>> - Is this a valid use case for CouchDB? I want to use it primarily
>> because
>>> of its good replication capabilities, especially in not reliable
>>> environments with some periods of being offline etc. Otherwise I'll have
>> to
>>> write the whole set of data sync APIs with buffering, retries etc myself.
>>> 
>> 
>> In our case, mobiles replicating to and from CouchDB, replication has
>> proven to be very very reliable, it just works.
>> If i had to implement that on my own it would have been much much worse.
>> 
>> 
>>> - Is this recommended practice to set up a chain of replication? Due to
>>> security considerations I want customer devices to replicate each to its
>>> own database in the cloud. Then I want those databases to replicate to
>> the
>>> single central log database I'd subscribe for _changes. The reason is
>> that
>>> it's easier for me to have a single source of _changes feed rather than
>>> multiple databases.
>>> 
>> 
>> We do this, each user got his own database which we consider "outside", we
>> monitor these databases for changes and take appropriate actions. :)
>> On our server continuous replications from thousands of customer databases
>> to a central database occupied to many connections¹ and the performance
>> over all degraded, even if only "some" users where actually active. We now
>> listen to DB changes (_db_updates), start replications for the DB in
>> question and stop them again after a certain timeout, activity obviously
>> resets the timeout.
>> 
>> If you consider the single central log database only so you have a single
>> changes feed (you don't need centralized views etc.) I would skip the
>> central database and just process _all_docs (or a view only containing
>> unprocessed logentries) of any database an "updated" event was triggered
>> on.
>> 
>> If you go that route, you are half way through rotating databases already,
>> your backend doesn't care anymore on which database a change is triggered
>> on.
>> 
>> 
>>> - Is using _purge safe in my case? From the official doc I read "In
>>> clustered or replicated environments it is very difficult to guarantee
>> that
>>> a particular purged document has been removed from all replicas". I don't
>>> think this is a problem for me as I primarily care about database size so
>>> it shouldn't be critical if some documents fail to delete.
>>> 
>> 
>> _purge is the wrong tool for this job.
>> From my understanding it's there as a last resort to get sensitive data out
>> of a DB.
>> 
>> Regards,
>> Stefan
>> 
>> [1] I think the main reason for this was actually the operating system, but
>> it was faster, easier and more future proof to implement the described
>> solution than to tune the OS to handle the connection, at least for me.
>> 
> -- 
> [image: Cabify - Your private Driver] <http://www.cabify.com/>
> 
> *Carlos Alonso*
> Data Engineer
> Madrid, Spain
> 
> carlos.alonso@cabify.com
> 
> Prueba gratis con este código
> #CARLOSA6319 <https://cabify.com/i/carlosa6319>
> [image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
> <http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
> Linkedin] <https://www.linkedin.com/in/mrcalonso>
> 
> -- 
> Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su 
> destinatario, pudiendo contener información confidencial sometida a secreto 
> profesional. No está permitida su reproducción o distribución sin la 
> autorización expresa de Cabify. Si usted no es el destinatario final por 
> favor elimínelo e infórmenos por esta vía. 
> 
> This message and any attached file are intended exclusively for the 
> addressee, and it may be confidential. You are not allowed to copy or 
> disclose it without Cabify's prior written authorization. If you are not 
> the intended recipient please delete it from your system and notify us by 
> e-mail.

Re: Using CouchDB for transferring user logs via the internet

[Carlos Alonso <ca...@cabify.com>]

Hi Vladimir.

I suppose you already have evaluated this option, but just in case... To me
this sounds like a task for a queue. Have you thought about having a queue
as your central logs storage (instead of the couchdb cluster on cloud you
described)? To me it feels a more natural use case. From your devices you
publish new messages (log entries) into the queue and then from your
application you consume those messages to process them. The queue will
already delete the processed messages for you, so you won't need to take
care of it (no rotating dbs, purging, ...). There are many different tools
out there so I guess you'll find one that provides your required
durability/ordering requirements out of the box....

Hope it helps

On Tue, Jun 20, 2017 at 11:12 PM Stefan Klein <st...@gmail.com> wrote:

> Hi,
>
> commenting on individual topics inline, though i only have experience with
> CouchDB 1.6, maybe CouchDB 2.0 behaves a bit different.
>
> 2017-06-20 20:34 GMT+02:00 Vladimir Kuznetsov <vo...@gmail.com>:
>
> >
> > Now, I know in couchdb documents are not being really deleted, just
> marked
> > as 'deleted' so the database will permanently grow. I've and option
> either
> > to use periodic _purge(which I heard may be not safe especially in
> > clustered environment) or implement this as monthly rotating
> database(which
> > is more complex and I don't really want to follow this route).
> >
>
> I think rotating databases are not that more complex, but see below.
>
>
> > My questions are:
> >
> > - Is this a valid use case for CouchDB? I want to use it primarily
> because
> > of its good replication capabilities, especially in not reliable
> > environments with some periods of being offline etc. Otherwise I'll have
> to
> > write the whole set of data sync APIs with buffering, retries etc myself.
> >
>
> In our case, mobiles replicating to and from CouchDB, replication has
> proven to be very very reliable, it just works.
> If i had to implement that on my own it would have been much much worse.
>
>
> > - Is this recommended practice to set up a chain of replication? Due to
> > security considerations I want customer devices to replicate each to its
> > own database in the cloud. Then I want those databases to replicate to
> the
> > single central log database I'd subscribe for _changes. The reason is
> that
> > it's easier for me to have a single source of _changes feed rather than
> > multiple databases.
> >
>
> We do this, each user got his own database which we consider "outside", we
> monitor these databases for changes and take appropriate actions. :)
> On our server continuous replications from thousands of customer databases
> to a central database occupied to many connections¹ and the performance
> over all degraded, even if only "some" users where actually active. We now
> listen to DB changes (_db_updates), start replications for the DB in
> question and stop them again after a certain timeout, activity obviously
> resets the timeout.
>
> If you consider the single central log database only so you have a single
> changes feed (you don't need centralized views etc.) I would skip the
> central database and just process _all_docs (or a view only containing
> unprocessed logentries) of any database an "updated" event was triggered
> on.
>
> If you go that route, you are half way through rotating databases already,
> your backend doesn't care anymore on which database a change is triggered
> on.
>
>
> > - Is using _purge safe in my case? From the official doc I read "In
> > clustered or replicated environments it is very difficult to guarantee
> that
> > a particular purged document has been removed from all replicas". I don't
> > think this is a problem for me as I primarily care about database size so
> > it shouldn't be critical if some documents fail to delete.
> >
>
> _purge is the wrong tool for this job.
> From my understanding it's there as a last resort to get sensitive data out
> of a DB.
>
> Regards,
> Stefan
>
> [1] I think the main reason for this was actually the operating system, but
> it was faster, easier and more future proof to implement the described
> solution than to tune the OS to handle the connection, at least for me.
>
-- 
[image: Cabify - Your private Driver] <http://www.cabify.com/>

*Carlos Alonso*
Data Engineer
Madrid, Spain

carlos.alonso@cabify.com

Prueba gratis con este código
#CARLOSA6319 <https://cabify.com/i/carlosa6319>
[image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
<http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
Linkedin] <https://www.linkedin.com/in/mrcalonso>

-- 
Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su 
destinatario, pudiendo contener información confidencial sometida a secreto 
profesional. No está permitida su reproducción o distribución sin la 
autorización expresa de Cabify. Si usted no es el destinatario final por 
favor elimínelo e infórmenos por esta vía. 

This message and any attached file are intended exclusively for the 
addressee, and it may be confidential. You are not allowed to copy or 
disclose it without Cabify's prior written authorization. If you are not 
the intended recipient please delete it from your system and notify us by 
e-mail.

Re: Using CouchDB for transferring user logs via the internet

[Stefan Klein <st...@gmail.com>]

Hi,

commenting on individual topics inline, though i only have experience with
CouchDB 1.6, maybe CouchDB 2.0 behaves a bit different.

2017-06-20 20:34 GMT+02:00 Vladimir Kuznetsov <vo...@gmail.com>:

>
> Now, I know in couchdb documents are not being really deleted, just marked
> as 'deleted' so the database will permanently grow. I've and option either
> to use periodic _purge(which I heard may be not safe especially in
> clustered environment) or implement this as monthly rotating database(which
> is more complex and I don't really want to follow this route).
>

I think rotating databases are not that more complex, but see below.


> My questions are:
>
> - Is this a valid use case for CouchDB? I want to use it primarily because
> of its good replication capabilities, especially in not reliable
> environments with some periods of being offline etc. Otherwise I'll have to
> write the whole set of data sync APIs with buffering, retries etc myself.
>

In our case, mobiles replicating to and from CouchDB, replication has
proven to be very very reliable, it just works.
If i had to implement that on my own it would have been much much worse.


> - Is this recommended practice to set up a chain of replication? Due to
> security considerations I want customer devices to replicate each to its
> own database in the cloud. Then I want those databases to replicate to the
> single central log database I'd subscribe for _changes. The reason is that
> it's easier for me to have a single source of _changes feed rather than
> multiple databases.
>

We do this, each user got his own database which we consider "outside", we
monitor these databases for changes and take appropriate actions. :)
On our server continuous replications from thousands of customer databases
to a central database occupied to many connections¹ and the performance
over all degraded, even if only "some" users where actually active. We now
listen to DB changes (_db_updates), start replications for the DB in
question and stop them again after a certain timeout, activity obviously
resets the timeout.

If you consider the single central log database only so you have a single
changes feed (you don't need centralized views etc.) I would skip the
central database and just process _all_docs (or a view only containing
unprocessed logentries) of any database an "updated" event was triggered on.

If you go that route, you are half way through rotating databases already,
your backend doesn't care anymore on which database a change is triggered
on.


> - Is using _purge safe in my case? From the official doc I read "In
> clustered or replicated environments it is very difficult to guarantee that
> a particular purged document has been removed from all replicas". I don't
> think this is a problem for me as I primarily care about database size so
> it shouldn't be critical if some documents fail to delete.
>

_purge is the wrong tool for this job.
From my understanding it's there as a last resort to get sensitive data out
of a DB.

Regards,
Stefan

[1] I think the main reason for this was actually the operating system, but
it was faster, easier and more future proof to implement the described
solution than to tune the OS to handle the connection, at least for me.