You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Maxim Osipov <ma...@flexibity.com> on 2012/06/08 13:08:01 UTC
versioning for sensors data storage
Hi,
Warning: I'm new to CouchDB! And I'm in a process pf selecting a
database to store data from sensors. The scenario looks like this:
There are sensors on the net (may be thousands or later millions) for
things like temperature, motion, etc. Each sensors has an IPv6 address
and pushes data to central server in JSON format, like:
{ temp: 36.6; humidity: 80 }
We don't know in advance what exactly sensor will push, but most of
the time the format of JSON structure is the same. Data is updated ...
say every second for each sensor.
>From the "architecture" point of view it could be a nice solution with
CouchDB, where each sensor is represented by a document and each
sample of sensor data is a version of this document.
The questions are:
- do we have time/date information for versions
- is it easy/good to access document version history
- what is a limit on number of versions and how it affects performance
- is it possible/easy/good to get a time snapshot of database
All other comments are also welcome!
Kind regards,
Maxim
--
Maxim Osipov
Flexibity Ltd.
Re: versioning for sensors data storage
Posted by Simon Metson <si...@cloudant.com>.
Hi,
> From the "architecture" point of view it could be a nice solution with
> CouchDB, where each sensor is represented by a document and each
> sample of sensor data is a version of this document.
>
>
You don't want to do that, as soon as a compaction occurs you'll loose your history. Instead make a document for each sample, and include the sensor id in that document, so you can build a view of the samples for a given sensor.
> The questions are:
> - do we have time/date information for versions
>
>
No, you'd need to add that into the document yourself.
> - is it easy/good to access document version history
> - what is a limit on number of versions and how it affects performance
>
>
See above
> - is it possible/easy/good to get a time snapshot of database
>
>
>
You mean for backup? Sure, either copy the db file or if you need to access it remotely you can page through _all_docs to get a snapshot. You could also use the _changes feed to do a rolling backup.
Cheers
Simon
Re: versioning for sensors data storage
Posted by david martin <da...@lymegreen.co.uk>.
On 08/06/12 12:08, Maxim Osipov wrote:
> Hi,
>
> Warning: I'm new to CouchDB! And I'm in a process pf selecting a
> database to store data from sensors. The scenario looks like this:
>
> There are sensors on the net (may be thousands or later millions) for
> things like temperature, motion, etc. Each sensors has an IPv6 address
> and pushes data to central server in JSON format, like:
> { temp: 36.6; humidity: 80 }
> We don't know in advance what exactly sensor will push, but most of
> the time the format of JSON structure is the same. Data is updated ...
> say every second for each sensor.
>
> > From the "architecture" point of view it could be a nice solution with
> CouchDB, where each sensor is represented by a document and each
> sample of sensor data is a version of this document.
Each sample time is represented by a document and each sensor has a field in the document.
If multiple sensors read at the same time different documents are produced for each sensor.
If sensors are all read at the same time multiple fields can exist in a single time document.
Versions have nothing to do with what you are trying to do.
>
> The questions are:
> - do we have time/date information for versions
not a viable idea
> - is it easy/good to access document version history
not for this purpose
> - what is a limit on number of versions and how it affects performance
arbitrary 1000, but not to be used for this purpose
> - is it possible/easy/good to get a time snapshot of database
yes
>
> All other comments are also welcome!
>
> Kind regards,
> Maxim
>
http://otperl.com/my_nanode
381153 minute readings of temperature data in Couchdb
visualized by couchapp running d3.js
data comes from Nanode #490
some issues with negative degree readings on Nanode.
press Next (not start) to visualize.
You cannot break anything!
records in JSON, date is coded in id
dt = new Date(parseInt(parseInt(json.rows[0].id.slice(0,14),16)/1000));
|{
"|_id|":|"04c1f8cbefdf8af6aa4f5cff8291a2fb"|,
"|_rev|":|"1-56da4c7f440fcd48fd60a222e502d30b"|,
"|Nanode|":|"5.0.0"|,
"|NodeNo|":|"0490"|,
"|comment|":|"comment"|,
"|datastreams|":|[
|{
"|id|":|"temperatureC"|,
"|value|":|"8.83"|
}|,
|{
"|id|":|"F"|,
"|value|":|"47.24"|
}|,
|{
"|id|":|"mv"|,
"|value|":|"0.58"|
}|
]|
}|
David Martin (davidoccam)
Re: versioning for sensors data storage
Posted by Robert Newson <ro...@gmail.com>.
user@ please.
On 8 Jun 2012, at 12:08, Maxim Osipov wrote:
> Hi,
>
> Warning: I'm new to CouchDB! And I'm in a process pf selecting a
> database to store data from sensors. The scenario looks like this:
>
> There are sensors on the net (may be thousands or later millions) for
> things like temperature, motion, etc. Each sensors has an IPv6 address
> and pushes data to central server in JSON format, like:
> { temp: 36.6; humidity: 80 }
> We don't know in advance what exactly sensor will push, but most of
> the time the format of JSON structure is the same. Data is updated ...
> say every second for each sensor.
>
> From the "architecture" point of view it could be a nice solution with
> CouchDB, where each sensor is represented by a document and each
> sample of sensor data is a version of this document.
>
> The questions are:
> - do we have time/date information for versions
> - is it easy/good to access document version history
> - what is a limit on number of versions and how it affects performance
> - is it possible/easy/good to get a time snapshot of database
>
> All other comments are also welcome!
>
> Kind regards,
> Maxim
>
> --
> Maxim Osipov
> Flexibity Ltd.