You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alejandro Calbazana <ac...@gmail.com> on 2013/10/30 15:10:28 UTC
Computing Results So That They are Returned in Search Results
I'd like to throw out a design question and see if its possible to solve
this with Solr.
I have a set of data that is computed that I'd like to make searchable.
Ideally, I'd like to have all documents indexed and call it the day, but
the nature of the data is such that it needs to be computed given a
definition. I'm interested in searching on definitions and then creating
results on the fly that are calculated based on something embedded in the
definition.
Is it possible to embed this calculation login into Solr's result handling
process? I know this sounds exotic, but the nature of the data is such
that I can't index these calculated documents because I don't know what the
boundary is and specifiying an arbitrary number isn't ideal.
Has anyone run across something like this?
Thanks,
Alejandr
Re: Computing Results So That They are Returned in Search Results
Posted by Alejandro Calbazana <ac...@gmail.com>.
So here is my use case with a little more detail. I'm working with
recurring events. Each event has an expression associated with it that
defines its recurrence pattern. For example, monthly, daily, yearly...
The event has metadata associated with it that is searchable. When a user
performs a search, they can match on various metadata fields, but the query
can also span a range of dates. If a match occurs, I'd like to unwind the
expression into the instances specified by the pattern and return these
"virtual" instances as results.
Right now, I'm post processing data to hammer out the results that fit the
window of time specified in the query, but this moves sorting and
pagination out of the Solr tier. I'd like to see if I can get it to stay
there :) Post processing also prohibits me from faceting which would be
extremely useful.
I'm trying to avoid heavy post processing if I can. Given the nature of
the data, its not really feasible for me to pre-assemble instance data and
index since I don't know the window of time a user will be looking at.
Thanks,
Alejandro
On Wed, Oct 30, 2013 at 6:35 PM, Upayavira <uv...@odoko.co.uk> wrote:
> Also note that function queries only return numbers (given their origin
> in scoring). They cannot be used to create virtual string or text
> fields.
>
> Upayavira
>
> On Wed, Oct 30, 2013, at 05:19 PM, Jack Krupansky wrote:
> > A function query is simply returning a calculated result based on
> > existing
> > data - no new fields required.
> >
> > Did you actually want to precompute a value, store it in the index, and
> > then
> > query on it? If so, you could do that indexing with a custom or scripted
> > update processor.
> >
> > Flesh out an example of exactly what you want.
> >
> > -- Jack Krupansky
> >
> > -----Original Message-----
> > From: Alejandro Calbazana
> > Sent: Wednesday, October 30, 2013 12:46 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Computing Results So That They are Returned in Search
> > Results
> >
> > Sounds really close to what I'm looking for, but this sounds like it
> > would
> > result in a new field on a document (or a new value for a field defined
> > to
> > hold the result of a function). Would it be possible for a function
> > query
> > to produce a new document so that I can associate the computed value with
> > it?
> >
> > Thanks,
> >
> > Alejandro
> >
> >
> > On Wed, Oct 30, 2013 at 12:05 PM, Jack Krupansky
> > <ja...@basetechnology.com>wrote:
> >
> > > You could create a custom "value source" and then use it in a function
> > > query embedded in your return fields list (fl).
> > >
> > > So, the function query could use a function (value source) that takes a
> > > field, fetches its value, performs some arbitrary calculation, and then
> > > returns that value.
> > >
> > > fl=id,name,my-func(field1),my-**func(field2)
> > >
> > > -- Jack Krupansky
> > >
> > > -----Original Message----- From: Alejandro Calbazana
> > > Sent: Wednesday, October 30, 2013 10:10 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Computing Results So That They are Returned in Search Results
> > >
> > > I'd like to throw out a design question and see if its possible to
> solve
> > > this with Solr.
> > >
> > > I have a set of data that is computed that I'd like to make searchable.
> > > Ideally, I'd like to have all documents indexed and call it the day,
> but
> > > the nature of the data is such that it needs to be computed given a
> > > definition. I'm interested in searching on definitions and then
> creating
> > > results on the fly that are calculated based on something embedded in
> the
> > > definition.
> > >
> > > Is it possible to embed this calculation login into Solr's result
> handling
> > > process? I know this sounds exotic, but the nature of the data is such
> > > that I can't index these calculated documents because I don't know what
> > > the
> > > boundary is and specifiying an arbitrary number isn't ideal.
> > >
> > > Has anyone run across something like this?
> > >
> > > Thanks,
> > >
> > > Alejandr
> > >
> >
>
Re: Computing Results So That They are Returned in Search Results
Posted by Upayavira <uv...@odoko.co.uk>.
Also note that function queries only return numbers (given their origin
in scoring). They cannot be used to create virtual string or text
fields.
Upayavira
On Wed, Oct 30, 2013, at 05:19 PM, Jack Krupansky wrote:
> A function query is simply returning a calculated result based on
> existing
> data - no new fields required.
>
> Did you actually want to precompute a value, store it in the index, and
> then
> query on it? If so, you could do that indexing with a custom or scripted
> update processor.
>
> Flesh out an example of exactly what you want.
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: Alejandro Calbazana
> Sent: Wednesday, October 30, 2013 12:46 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Computing Results So That They are Returned in Search
> Results
>
> Sounds really close to what I'm looking for, but this sounds like it
> would
> result in a new field on a document (or a new value for a field defined
> to
> hold the result of a function). Would it be possible for a function
> query
> to produce a new document so that I can associate the computed value with
> it?
>
> Thanks,
>
> Alejandro
>
>
> On Wed, Oct 30, 2013 at 12:05 PM, Jack Krupansky
> <ja...@basetechnology.com>wrote:
>
> > You could create a custom "value source" and then use it in a function
> > query embedded in your return fields list (fl).
> >
> > So, the function query could use a function (value source) that takes a
> > field, fetches its value, performs some arbitrary calculation, and then
> > returns that value.
> >
> > fl=id,name,my-func(field1),my-**func(field2)
> >
> > -- Jack Krupansky
> >
> > -----Original Message----- From: Alejandro Calbazana
> > Sent: Wednesday, October 30, 2013 10:10 AM
> > To: solr-user@lucene.apache.org
> > Subject: Computing Results So That They are Returned in Search Results
> >
> > I'd like to throw out a design question and see if its possible to solve
> > this with Solr.
> >
> > I have a set of data that is computed that I'd like to make searchable.
> > Ideally, I'd like to have all documents indexed and call it the day, but
> > the nature of the data is such that it needs to be computed given a
> > definition. I'm interested in searching on definitions and then creating
> > results on the fly that are calculated based on something embedded in the
> > definition.
> >
> > Is it possible to embed this calculation login into Solr's result handling
> > process? I know this sounds exotic, but the nature of the data is such
> > that I can't index these calculated documents because I don't know what
> > the
> > boundary is and specifiying an arbitrary number isn't ideal.
> >
> > Has anyone run across something like this?
> >
> > Thanks,
> >
> > Alejandr
> >
>
Re: Computing Results So That They are Returned in Search Results
Posted by Jack Krupansky <ja...@basetechnology.com>.
A function query is simply returning a calculated result based on existing
data - no new fields required.
Did you actually want to precompute a value, store it in the index, and then
query on it? If so, you could do that indexing with a custom or scripted
update processor.
Flesh out an example of exactly what you want.
-- Jack Krupansky
-----Original Message-----
From: Alejandro Calbazana
Sent: Wednesday, October 30, 2013 12:46 PM
To: solr-user@lucene.apache.org
Subject: Re: Computing Results So That They are Returned in Search Results
Sounds really close to what I'm looking for, but this sounds like it would
result in a new field on a document (or a new value for a field defined to
hold the result of a function). Would it be possible for a function query
to produce a new document so that I can associate the computed value with
it?
Thanks,
Alejandro
On Wed, Oct 30, 2013 at 12:05 PM, Jack Krupansky
<ja...@basetechnology.com>wrote:
> You could create a custom "value source" and then use it in a function
> query embedded in your return fields list (fl).
>
> So, the function query could use a function (value source) that takes a
> field, fetches its value, performs some arbitrary calculation, and then
> returns that value.
>
> fl=id,name,my-func(field1),my-**func(field2)
>
> -- Jack Krupansky
>
> -----Original Message----- From: Alejandro Calbazana
> Sent: Wednesday, October 30, 2013 10:10 AM
> To: solr-user@lucene.apache.org
> Subject: Computing Results So That They are Returned in Search Results
>
> I'd like to throw out a design question and see if its possible to solve
> this with Solr.
>
> I have a set of data that is computed that I'd like to make searchable.
> Ideally, I'd like to have all documents indexed and call it the day, but
> the nature of the data is such that it needs to be computed given a
> definition. I'm interested in searching on definitions and then creating
> results on the fly that are calculated based on something embedded in the
> definition.
>
> Is it possible to embed this calculation login into Solr's result handling
> process? I know this sounds exotic, but the nature of the data is such
> that I can't index these calculated documents because I don't know what
> the
> boundary is and specifiying an arbitrary number isn't ideal.
>
> Has anyone run across something like this?
>
> Thanks,
>
> Alejandr
>
Re: Computing Results So That They are Returned in Search Results
Posted by Alejandro Calbazana <ac...@gmail.com>.
Sounds really close to what I'm looking for, but this sounds like it would
result in a new field on a document (or a new value for a field defined to
hold the result of a function). Would it be possible for a function query
to produce a new document so that I can associate the computed value with
it?
Thanks,
Alejandro
On Wed, Oct 30, 2013 at 12:05 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
> You could create a custom "value source" and then use it in a function
> query embedded in your return fields list (fl).
>
> So, the function query could use a function (value source) that takes a
> field, fetches its value, performs some arbitrary calculation, and then
> returns that value.
>
> fl=id,name,my-func(field1),my-**func(field2)
>
> -- Jack Krupansky
>
> -----Original Message----- From: Alejandro Calbazana
> Sent: Wednesday, October 30, 2013 10:10 AM
> To: solr-user@lucene.apache.org
> Subject: Computing Results So That They are Returned in Search Results
>
> I'd like to throw out a design question and see if its possible to solve
> this with Solr.
>
> I have a set of data that is computed that I'd like to make searchable.
> Ideally, I'd like to have all documents indexed and call it the day, but
> the nature of the data is such that it needs to be computed given a
> definition. I'm interested in searching on definitions and then creating
> results on the fly that are calculated based on something embedded in the
> definition.
>
> Is it possible to embed this calculation login into Solr's result handling
> process? I know this sounds exotic, but the nature of the data is such
> that I can't index these calculated documents because I don't know what the
> boundary is and specifiying an arbitrary number isn't ideal.
>
> Has anyone run across something like this?
>
> Thanks,
>
> Alejandr
>
Re: Computing Results So That They are Returned in Search Results
Posted by Jack Krupansky <ja...@basetechnology.com>.
You could create a custom "value source" and then use it in a function query
embedded in your return fields list (fl).
So, the function query could use a function (value source) that takes a
field, fetches its value, performs some arbitrary calculation, and then
returns that value.
fl=id,name,my-func(field1),my-func(field2)
-- Jack Krupansky
-----Original Message-----
From: Alejandro Calbazana
Sent: Wednesday, October 30, 2013 10:10 AM
To: solr-user@lucene.apache.org
Subject: Computing Results So That They are Returned in Search Results
I'd like to throw out a design question and see if its possible to solve
this with Solr.
I have a set of data that is computed that I'd like to make searchable.
Ideally, I'd like to have all documents indexed and call it the day, but
the nature of the data is such that it needs to be computed given a
definition. I'm interested in searching on definitions and then creating
results on the fly that are calculated based on something embedded in the
definition.
Is it possible to embed this calculation login into Solr's result handling
process? I know this sounds exotic, but the nature of the data is such
that I can't index these calculated documents because I don't know what the
boundary is and specifiying an arbitrary number isn't ideal.
Has anyone run across something like this?
Thanks,
Alejandr