Conditional/Calculated Fields (is it possible?)

[Andrew Ingram <an...@andrewingram.net>]

Hi everyone,

I'm working on the search schema for ecommerce products and I'm having
an issue with the prices.

Basically, a product has two price values and a date, the product
effectively has one price before the date and the other one after.
This poses no problem for the site itself since I can use conditional
logic, but I have no idea how to approach this with regards to solr
queries.

The price of a product is used for both faceting and sorting and
should use whichever price is active at the time of the query. Is
there any way to do define a field whose value is a simple algorithm
operating on the value of other fields?

I'm quite happy to use a custom field type if necessary, though I'm
not sure if what I want is even possible and I don't really know where
to begin.

Any help would be appreciated

Regards,
Andrew Ingram

Re: Conditional/Calculated Fields (is it possible?)

[Avlesh Singh <av...@gmail.com>]

How about this -
Lets say your domain objects had these three things you care about - P1, D
and P2 (P1 is the price valid before D and P2 after this date).
Index each such document twice (Please don't kill me for suggesting this).
There are two fields which you should index with the document - epoch and
price. One document has a value D in the epoch field and price P1. The other
document would have a value (D+1) in the epoch field and P2 in the price
field.

I am assuming, that all your queries to this index have a filterQuery
appended to it, always. So for any given date that you want to query on, you
can add a filter query (epoch lt incomingDate) to your query. This will make
sure you always get just one document of the two that has been indexed.

Other than doubling your index size, is there something fundamentally wrong
with this design?
More importantly, does this work?

Cheers
Avlesh

On Thu, May 7, 2009 at 6:15 AM, Yonik Seeley <yo...@lucidimagination.com>wrote:

> On Wed, May 6, 2009 at 7:32 AM, Andrew Ingram <an...@andrewingram.net>
> wrote:
> > Basically, a product has two price values and a date, the product
> > effectively has one price before the date and the other one after.
> > This poses no problem for the site itself since I can use conditional
> > logic, but I have no idea how to approach this with regards to solr
> > queries.
> >
> > The price of a product is used for both faceting and sorting and
> > should use whichever price is active at the time of the query. Is
> > there any way to do define a field whose value is a simple algorithm
> > operating on the value of other fields?
>
> Sort of - function queries.
>
> We don't have any conditional function queries yet, but that's easy.
> You can't yet sort by a function query, but we should fix that.
> You can't yet facet on a function query, but we should fix that too.
>
> Short term, Erick's index the doc twice idea may be best.
> Long term, out-of-the-box functionality after volunteers help develop
> out the function query approach :-)
>
> -Yonik
> http://www.lucidimagination.com
>

Re: Conditional/Calculated Fields (is it possible?)

[Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>]

On Thu, May 7, 2009 at 6:15 AM, Yonik Seeley <yo...@lucidimagination.com> wrote:
> On Wed, May 6, 2009 at 7:32 AM, Andrew Ingram <an...@andrewingram.net> wrote:
>> Basically, a product has two price values and a date, the product
>> effectively has one price before the date and the other one after.
>> This poses no problem for the site itself since I can use conditional
>> logic, but I have no idea how to approach this with regards to solr
>> queries.
>>
>> The price of a product is used for both faceting and sorting and
>> should use whichever price is active at the time of the query. Is
>> there any way to do define a field whose value is a simple algorithm
>> operating on the value of other fields?
>
> Sort of - function queries.
>
> We don't have any conditional function queries yet, but that's easy.
> You can't yet sort by a function query, but we should fix that.
> You can't yet facet on a function query, but we should fix that too.

+1 we should fix this
>
> Short term, Erick's index the doc twice idea may be best.
> Long term, out-of-the-box functionality after volunteers help develop
> out the function query approach :-)

Ericks's idea is the simplest for now. Good one
>
> -Yonik
> http://www.lucidimagination.com
>



-- 
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: Conditional/Calculated Fields (is it possible?)

[Yonik Seeley <yo...@lucidimagination.com>]

On Wed, May 6, 2009 at 7:32 AM, Andrew Ingram <an...@andrewingram.net> wrote:
> Basically, a product has two price values and a date, the product
> effectively has one price before the date and the other one after.
> This poses no problem for the site itself since I can use conditional
> logic, but I have no idea how to approach this with regards to solr
> queries.
>
> The price of a product is used for both faceting and sorting and
> should use whichever price is active at the time of the query. Is
> there any way to do define a field whose value is a simple algorithm
> operating on the value of other fields?

Sort of - function queries.

We don't have any conditional function queries yet, but that's easy.
You can't yet sort by a function query, but we should fix that.
You can't yet facet on a function query, but we should fix that too.

Short term, Erick's index the doc twice idea may be best.
Long term, out-of-the-box functionality after volunteers help develop
out the function query approach :-)

-Yonik
http://www.lucidimagination.com

Re: Conditional/Calculated Fields (is it possible?)

[Andrew Ingram <an...@andrewingram.net>]

That is *very* clever and will almost certainly work. My only concern is 
that it doubles the overhead (and has a not insignificant effect on the 
complexity of my code) for what is essentially a minor feature. I'd like 
to keep this approach on the back-burner and see if a more conventional 
solution exists. Very nice idea though.

- Andy

Erick Erickson wrote:
> Would it work to index the item N times, where N is the number
> of discrete date ranges?
>
> Say you had a hat that was $10 from 1/1/09 to 6/30/09 and $11 from
> 7/1/09 to 12/30/09. Just index it twice with the dates something like
> valid_from, valid_to. Then form your queries so you were looking
> in the proper date...
>
> No idea if this *really* helps or not.
>
> Best
> Erick
>
> On Wed, May 6, 2009 at 7:32 AM, Andrew Ingram <an...@andrewingram.net> wrote:
>
>   
>> Hi everyone,
>>
>> I'm working on the search schema for ecommerce products and I'm having
>> an issue with the prices.
>>
>> Basically, a product has two price values and a date, the product
>> effectively has one price before the date and the other one after.
>> This poses no problem for the site itself since I can use conditional
>> logic, but I have no idea how to approach this with regards to solr
>> queries.
>>
>> The price of a product is used for both faceting and sorting and
>> should use whichever price is active at the time of the query. Is
>> there any way to do define a field whose value is a simple algorithm
>> operating on the value of other fields?
>>
>> I'm quite happy to use a custom field type if necessary, though I'm
>> not sure if what I want is even possible and I don't really know where
>> to begin.
>>
>> Any help would be appreciated
>>
>> Regards,
>> Andrew Ingram
>>
>>     
>
>   

Re: Conditional/Calculated Fields (is it possible?)

[Erick Erickson <er...@gmail.com>]

Would it work to index the item N times, where N is the number
of discrete date ranges?

Say you had a hat that was $10 from 1/1/09 to 6/30/09 and $11 from
7/1/09 to 12/30/09. Just index it twice with the dates something like
valid_from, valid_to. Then form your queries so you were looking
in the proper date...

No idea if this *really* helps or not.

Best
Erick

On Wed, May 6, 2009 at 7:32 AM, Andrew Ingram <an...@andrewingram.net> wrote:

> Hi everyone,
>
> I'm working on the search schema for ecommerce products and I'm having
> an issue with the prices.
>
> Basically, a product has two price values and a date, the product
> effectively has one price before the date and the other one after.
> This poses no problem for the site itself since I can use conditional
> logic, but I have no idea how to approach this with regards to solr
> queries.
>
> The price of a product is used for both faceting and sorting and
> should use whichever price is active at the time of the query. Is
> there any way to do define a field whose value is a simple algorithm
> operating on the value of other fields?
>
> I'm quite happy to use a custom field type if necessary, though I'm
> not sure if what I want is even possible and I don't really know where
> to begin.
>
> Any help would be appreciated
>
> Regards,
> Andrew Ingram
>

Re: Conditional/Calculated Fields (is it possible?)

[Andrew Ingram <an...@andrewingram.net>]

Hi Grant,

The problem is that even if we're only displaying 16 products at a time, 
we're potentially sorting over a collection of thousands. This means 
it's pretty essential for the ordering from solr to be correct.

- Andy

Grant Ingersoll wrote:
> WARNING: Low level Solr hacking and brainstorming:
> I _wonder_ if you could plug in a FieldSelector that was aware of the 
> date and then made the appropriate decision to return the correct 
> price field.  (On a side note, I'd love to see Solr support generic 
> FieldSelector plugin capability)
>
>
> Alternatively, you could right a simple "post-processing" 
> SearchComponent that loops over the return doc list (not the full doc 
> set) and added a field to each document that contained the "official 
> price".  Assuming you are only returning 10 or so results, I would 
> think this would still perform pretty well.
>
>
>
> -Grant
>
> On May 6, 2009, at 4:32 AM, Andrew Ingram wrote:
>
>> Hi everyone,
>>
>> I'm working on the search schema for ecommerce products and I'm having
>> an issue with the prices.
>>
>> Basically, a product has two price values and a date, the product
>> effectively has one price before the date and the other one after.
>> This poses no problem for the site itself since I can use conditional
>> logic, but I have no idea how to approach this with regards to solr
>> queries.
>>
>> The price of a product is used for both faceting and sorting and
>> should use whichever price is active at the time of the query. Is
>> there any way to do define a field whose value is a simple algorithm
>> operating on the value of other fields?
>>
>> I'm quite happy to use a custom field type if necessary, though I'm
>> not sure if what I want is even possible and I don't really know where
>> to begin.
>>
>> Any help would be appreciated
>>
>> Regards,
>> Andrew Ingram
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) 
> using Solr/Lucene:
> http://www.lucidimagination.com/search
>
>

Re: Conditional/Calculated Fields (is it possible?)

[Grant Ingersoll <gs...@apache.org>]

WARNING: Low level Solr hacking and brainstorming:
I _wonder_ if you could plug in a FieldSelector that was aware of the  
date and then made the appropriate decision to return the correct  
price field.  (On a side note, I'd love to see Solr support generic  
FieldSelector plugin capability)


Alternatively, you could right a simple "post-processing"  
SearchComponent that loops over the return doc list (not the full doc  
set) and added a field to each document that contained the "official  
price".  Assuming you are only returning 10 or so results, I would  
think this would still perform pretty well.



-Grant

On May 6, 2009, at 4:32 AM, Andrew Ingram wrote:

> Hi everyone,
>
> I'm working on the search schema for ecommerce products and I'm having
> an issue with the prices.
>
> Basically, a product has two price values and a date, the product
> effectively has one price before the date and the other one after.
> This poses no problem for the site itself since I can use conditional
> logic, but I have no idea how to approach this with regards to solr
> queries.
>
> The price of a product is used for both faceting and sorting and
> should use whichever price is active at the time of the query. Is
> there any way to do define a field whose value is a simple algorithm
> operating on the value of other fields?
>
> I'm quite happy to use a custom field type if necessary, though I'm
> not sure if what I want is even possible and I don't really know where
> to begin.
>
> Any help would be appreciated
>
> Regards,
> Andrew Ingram

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:
http://www.lucidimagination.com/search

Re: Conditional/Calculated Fields (is it possible?)

[Matt Weber <ma...@mattweber.org>]

I do not think this is possible.  You will probably want to handle  
this logic on your side during indexing.  Index the document with the  
fist price, then as that price expires, update the document with the  
new price.

Thanks,

Matt Weber
eSr Technologies
http://www.esr-technologies.com




On May 6, 2009, at 4:32 AM, Andrew Ingram wrote:

> Hi everyone,
>
> I'm working on the search schema for ecommerce products and I'm having
> an issue with the prices.
>
> Basically, a product has two price values and a date, the product
> effectively has one price before the date and the other one after.
> This poses no problem for the site itself since I can use conditional
> logic, but I have no idea how to approach this with regards to solr
> queries.
>
> The price of a product is used for both faceting and sorting and
> should use whichever price is active at the time of the query. Is
> there any way to do define a field whose value is a simple algorithm
> operating on the value of other fields?
>
> I'm quite happy to use a custom field type if necessary, though I'm
> not sure if what I want is even possible and I don't really know where
> to begin.
>
> Any help would be appreciated
>
> Regards,
> Andrew Ingram