You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alan Miller <al...@gmail.com> on 2011/12/02 19:33:36 UTC
Multivalued field
Hi I have a webapp that plots a bunch of time series
Data which are just doubles coupled with a timestamp
Every chart in my webapp has a reportid in my db and i am wondering if it would be effective to usr solr to serve the data th my app instead of keeping the data in my rdbms.
Currently im using hadoop to calc and generate the report data and the sticking it in my rdbms but i could use solrj client to upload the data to a solr index
I know solr if for indexing text documents but would it be effective to use solr in this way?
I want to query by reportid and get back a series of timestamp:double pairs.
Regards
Alan
Re: Multivalued field
Posted by Erick Erickson <er...@gmail.com>.
<field name="id" type="string" stored="true" indexed="true" required="true" />
<field name="data" type="text_en" stored="true" indexed="false" />
Then sometime later
<uniqueKey>id</uniqueKey>
(all this in your schema.xml file).
That's it. The data field isn't analyzed at all, so the type is largely
irrelevant. what you put in it is all your pairs of doubles in some
kind of delimited format, e.g. 2345.0,<timestamp> | 873945.7,<timestamp>
Now you just get your data field back, split it up and go.
Getting the report document will be about as fast as anything you could
do in Solr, lookup by what is essentially the primary key.
Updating your reports is just re-indexing (use the timestamp in your
DB) and it'll automatically replace documents with the same
id.
You *might* be able to use the "binary" type, but that's base64 encoded
so whether it would be faster than parsing your pairs from text
is an open question.
But what's really unclear is how ginormous your double/timestamp pairs
are. If you're pulling a billion pairs out, Solr performance won't be
your problem <G>....
Best
Erick
On Mon, Dec 5, 2011 at 2:24 PM, Alan Miller <al...@gmail.com> wrote:
>
> I know I'm using SolR for a task that is better suited for the DB to handle but I'm
> doing this for reasons related to the overall design of my system. My DB is going to
> become very large over time and it is constantly being updated via Hadoop jobs that
> collect,analyze some data and generate the final (report) results.
>
> The front end web-app needs to be VERY fast and only needs access to a subset of the data.
> It also let's us decouple the state of the DB and the front end, ie we can control when we sync
> the data from the DB to the SolR indexes.
> You could say I'm using SolR as an in memory cache of my DB indexes.
>
> We're also a small team and all our development is in java Hadoop, GWT so it was very
> easy for us to integrate SolR and Solrj into our app.
>
> If somebody could toss in an example of what the scheme might look like that'd be great.
> I have a very simple VALUE table that has columns:
> value_pk INTEGER ; primary-key
> report_fk INT ; foreign-key to report table
> tstamp TIMESTAMP
> value NUMERIC(7,4)
>
> Alan
>
> On Dec 5, 2011, at 14:34, Erick Erickson <er...@gmail.com> wrote:
>
>> Well, Solr is a text search engine, and a good one. But this sure
>> feels like a problem that RDBMSs were built to handle. Why do
>> you want to do this? Is your current performance a problem?
>> Are you blowing your space resources out of the water? Do you
>> want to distribute your app to places not connected to your RDBMS?
>> Is there too much traffic on your RDBMS machine?
>>
>> Something about "if it ain't broke, don't fix it".
>>
>> In general, you have to tell us the problem you're trying to solve
>> so we don't go off into XY land.
>> http://people.apache.org/~hossman/#xyproblem
>>
>> Best
>> Erick
>>
>> On Fri, Dec 2, 2011 at 1:33 PM, Alan Miller <al...@gmail.com> wrote:
>>> Hi I have a webapp that plots a bunch of time series
>>> Data which are just doubles coupled with a timestamp
>>>
>>> Every chart in my webapp has a reportid in my db and i am wondering if it would be effective to usr solr to serve the data th my app instead of keeping the data in my rdbms.
>>>
>>> Currently im using hadoop to calc and generate the report data and the sticking it in my rdbms but i could use solrj client to upload the data to a solr index
>>>
>>> I know solr if for indexing text documents but would it be effective to use solr in this way?
>>>
>>> I want to query by reportid and get back a series of timestamp:double pairs.
>>>
>>> Regards
>>> Alan
Re: Multivalued field
Posted by Alan Miller <al...@gmail.com>.
I know I'm using SolR for a task that is better suited for the DB to handle but I'm
doing this for reasons related to the overall design of my system. My DB is going to
become very large over time and it is constantly being updated via Hadoop jobs that
collect,analyze some data and generate the final (report) results.
The front end web-app needs to be VERY fast and only needs access to a subset of the data.
It also let's us decouple the state of the DB and the front end, ie we can control when we sync
the data from the DB to the SolR indexes.
You could say I'm using SolR as an in memory cache of my DB indexes.
We're also a small team and all our development is in java Hadoop, GWT so it was very
easy for us to integrate SolR and Solrj into our app.
If somebody could toss in an example of what the scheme might look like that'd be great.
I have a very simple VALUE table that has columns:
value_pk INTEGER ; primary-key
report_fk INT ; foreign-key to report table
tstamp TIMESTAMP
value NUMERIC(7,4)
Alan
On Dec 5, 2011, at 14:34, Erick Erickson <er...@gmail.com> wrote:
> Well, Solr is a text search engine, and a good one. But this sure
> feels like a problem that RDBMSs were built to handle. Why do
> you want to do this? Is your current performance a problem?
> Are you blowing your space resources out of the water? Do you
> want to distribute your app to places not connected to your RDBMS?
> Is there too much traffic on your RDBMS machine?
>
> Something about "if it ain't broke, don't fix it".
>
> In general, you have to tell us the problem you're trying to solve
> so we don't go off into XY land.
> http://people.apache.org/~hossman/#xyproblem
>
> Best
> Erick
>
> On Fri, Dec 2, 2011 at 1:33 PM, Alan Miller <al...@gmail.com> wrote:
>> Hi I have a webapp that plots a bunch of time series
>> Data which are just doubles coupled with a timestamp
>>
>> Every chart in my webapp has a reportid in my db and i am wondering if it would be effective to usr solr to serve the data th my app instead of keeping the data in my rdbms.
>>
>> Currently im using hadoop to calc and generate the report data and the sticking it in my rdbms but i could use solrj client to upload the data to a solr index
>>
>> I know solr if for indexing text documents but would it be effective to use solr in this way?
>>
>> I want to query by reportid and get back a series of timestamp:double pairs.
>>
>> Regards
>> Alan
Re: Multivalued field
Posted by Erick Erickson <er...@gmail.com>.
Well, Solr is a text search engine, and a good one. But this sure
feels like a problem that RDBMSs were built to handle. Why do
you want to do this? Is your current performance a problem?
Are you blowing your space resources out of the water? Do you
want to distribute your app to places not connected to your RDBMS?
Is there too much traffic on your RDBMS machine?
Something about "if it ain't broke, don't fix it".
In general, you have to tell us the problem you're trying to solve
so we don't go off into XY land.
http://people.apache.org/~hossman/#xyproblem
Best
Erick
On Fri, Dec 2, 2011 at 1:33 PM, Alan Miller <al...@gmail.com> wrote:
> Hi I have a webapp that plots a bunch of time series
> Data which are just doubles coupled with a timestamp
>
> Every chart in my webapp has a reportid in my db and i am wondering if it would be effective to usr solr to serve the data th my app instead of keeping the data in my rdbms.
>
> Currently im using hadoop to calc and generate the report data and the sticking it in my rdbms but i could use solrj client to upload the data to a solr index
>
> I know solr if for indexing text documents but would it be effective to use solr in this way?
>
> I want to query by reportid and get back a series of timestamp:double pairs.
>
> Regards
> Alan