SolR for time-series data

[Alan Miller <al...@gmail.com>]

Hi,

I have a webapp that plots a bunch of time series data which
is just a series of doubles coupled with a timestamp.

Every chart in my webapp has a chart_id in my db and i am wondering if it
would be
effective to usr solr to serve the data to my app instead of keeping the
data in my rdbms.

Currently I'm using hadoop to calc and generate the report data and the
sticking it in my
rdbms but I could use solrj client to upload the data to a solr index
directly.

I know solr if for indexing text documents but would it be effective to use
solr in this way?

I want to query by chart_id and get back a series of timestamp:double pairs.

Regards
Alan

Re: SolR for time-series data

[Óscar Marín Miró <os...@gmail.com>]

Hi Alan, at my job we had a really succesful implementation similar to what
you are proposing. With a classic RDBM, we hit serious performance issues
so, we moved to solr to display time series of data. The 'trick' was to
facet on a date field, to get 'counts' of data for a time series on a
specific field but don't know if you could be more specific about your data
schema to see it this applies.

Kind regards,

Oscar

On Sat, Dec 3, 2011 at 6:10 PM, Alan Miller <al...@gmail.com> wrote:

> Hi,
>
> I have a webapp that plots a bunch of time series data which
> is just a series of doubles coupled with a timestamp.
>
> Every chart in my webapp has a chart_id in my db and i am wondering if it
> would be
> effective to usr solr to serve the data to my app instead of keeping the
> data in my rdbms.
>
> Currently I'm using hadoop to calc and generate the report data and the
> sticking it in my
> rdbms but I could use solrj client to upload the data to a solr index
> directly.
>
> I know solr if for indexing text documents but would it be effective to use
> solr in this way?
>
> I want to query by chart_id and get back a series of timestamp:double
> pairs.
>
> Regards
> Alan
>



-- 
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
 --Roy H. William

Re: SolR for time-series data

[Michael Kuhlmann <ku...@solarier.de>]

Hi Alan,

Solr can do this fast and easy, but I wonder if a simple key-value-store 
won't fit better for your suits.

Do you really only need to query be chart_id, or do you also need to 
query by time range?

In either case, as long as your data fits into an in-memory database, I 
would suggest Redis to you. It's easy to install and use, and it's fast 
as hell.

If you want to query by time ranges, you can use lists and query them by 
range using lrange (http://www.redis.io/commands/lrange), at least when 
you know the first timestamp and the steps are even. Or use a sorted 
set, and make sure that the values differ.

In my opinion, Solr has too many features that you don't need.

-Kuli

Am 03.12.2011 18:10, schrieb Alan Miller:
> Hi,
>
> I have a webapp that plots a bunch of time series data which
> is just a series of doubles coupled with a timestamp.
>
> Every chart in my webapp has a chart_id in my db and i am wondering if it
> would be
> effective to usr solr to serve the data to my app instead of keeping the
> data in my rdbms.
>
> Currently I'm using hadoop to calc and generate the report data and the
> sticking it in my
> rdbms but I could use solrj client to upload the data to a solr index
> directly.
>
> I know solr if for indexing text documents but would it be effective to use
> solr in this way?
>
> I want to query by chart_id and get back a series of timestamp:double pairs.
>
> Regards
> Alan
>

Re: SolR for time-series data

[Ted Dunning <te...@gmail.com>]

Sax is attractive, but I have found it lacking in practice.  My primary
issue is that in order to get sufficient recall for practical matching
problems, I had to do enough query expansion that the speed advantage of
inverted indexes went away.

The OP was asking for blob storage, however, and I think that SolR is fine
for that.

There is also the question of access to time series based on annotations
produced by other programs.  If the annotations express your intent, then
SolR wins again.  IF the annotations are sax annotations and that works for
you, great, but I wouldn't be optimistic that this would handle a wide
range of time series problems.

On Sun, Dec 4, 2011 at 5:14 AM, Grant Ingersoll <gs...@apache.org> wrote:

> Definitely should be possible.  As an aside, I've also thought one could
> do more time series stuff.  Have a look at the iSax stuff by Shieh and
> Koegh: http://www.cs.ucr.edu/~eamonn/iSAX/iSAX.html
>
>
> On Dec 3, 2011, at 12:10 PM, Alan Miller wrote:
>
> > Hi,
> >
> > I have a webapp that plots a bunch of time series data which
> > is just a series of doubles coupled with a timestamp.
> >
> > Every chart in my webapp has a chart_id in my db and i am wondering if it
> > would be
> > effective to usr solr to serve the data to my app instead of keeping the
> > data in my rdbms.
> >
> > Currently I'm using hadoop to calc and generate the report data and the
> > sticking it in my
> > rdbms but I could use solrj client to upload the data to a solr index
> > directly.
> >
> > I know solr if for indexing text documents but would it be effective to
> use
> > solr in this way?
> >
> > I want to query by chart_id and get back a series of timestamp:double
> pairs.
> >
> > Regards
> > Alan
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
>

Re: SolR for time-series data

[Grant Ingersoll <gs...@apache.org>]

Definitely should be possible.  As an aside, I've also thought one could do more time series stuff.  Have a look at the iSax stuff by Shieh and Koegh: http://www.cs.ucr.edu/~eamonn/iSAX/iSAX.html


On Dec 3, 2011, at 12:10 PM, Alan Miller wrote:

> Hi,
> 
> I have a webapp that plots a bunch of time series data which
> is just a series of doubles coupled with a timestamp.
> 
> Every chart in my webapp has a chart_id in my db and i am wondering if it
> would be
> effective to usr solr to serve the data to my app instead of keeping the
> data in my rdbms.
> 
> Currently I'm using hadoop to calc and generate the report data and the
> sticking it in my
> rdbms but I could use solrj client to upload the data to a solr index
> directly.
> 
> I know solr if for indexing text documents but would it be effective to use
> solr in this way?
> 
> I want to query by chart_id and get back a series of timestamp:double pairs.
> 
> Regards
> Alan

--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com