You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by Grant Ingersoll <gs...@apache.org> on 2009/12/10 18:37:50 UTC
SOLR-1131: Poly Fields and ValueSource
OK, next up:
How to handle the need to create multiple ValueSource instances for a given poly field? FieldType.getValueSource() only returns a single instance.
I think there are a few options:
1. Change the signature above to return a list of ValueSources. This likely has back compat. issues.
2. Just change those functions that need to handle poly fields (the distance functions) to worry about them and realize that the other functions won't work with them
3. Punt and require the user to know the secret handshake that is the naming convention used. In some regard, this is no different than what any developer would experience now when using dynamic fields with function queries.
4. Others?
-Grant
Re: SOLR-1131: Poly Fields and ValueSource
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Dec 10, 2009 at 1:44 PM, Grant Ingersoll <gs...@apache.org> wrote:
[...]
> Maybe, there are tradeoffs though.
>
> Let's get concrete and look at the VectorDistanceFunction (dist()). It can currently take in an even number of ValueSource instances
Geo will be the dominant use of points by far... so we shouldn't
compromise much on the usability for geo when trying to generalize to
n-dimentional points.
> My inclination is to fudge on this and do something in ValueSourceParser for each of the functions that can deal w/ poly fields
Right. The higher level client (HTTP) interface is of primary importance.
> Of course, this requires documentation, etc. for others to be able to do the same for their custom Functions
The ability of others to do it is nice, but is a separate feature
(depending on how it's implemented it may not be public).
-Yonik
http://www.lucidimaginatin.com
Re: SOLR-1131: Poly Fields and ValueSource
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 10, 2009, at 1:54 PM, Grant Ingersoll wrote:
>
> D'oh, I see another way of doing this, namely the Distance functions only work with points.
>
> Namely, the second case above becomes:
> dist(2, point1, point(x1, y1));
So, I went this route and I think it actually plays out quite nice (in fact, I think it gives new options for other types of functions too!) Patch soon.
Re: SOLR-1131: Poly Fields and ValueSource
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 10, 2009, at 1:44 PM, Grant Ingersoll wrote:
>
> On Dec 10, 2009, at 1:26 PM, Yonik Seeley wrote:
>
>> I wouldn't necessarily link FieldType.isPolyField() to the idea of a
>> poly value source... they are two different things.
>
> Yep. The word Poly is overloaded here to mean multiple ValueSources, but it isn't necessarily tied to there being a poly field, even though a PolyField likely would create a PolyValueSource
>
>> For example, if NumericField had not already been written in Lucene, I
>> would have perhaps just indexed both the lat and lon into the same
>> lucene field. That part can be more of an implementation detail, and
>> does not reflect the semantics of the field (the fact that it contains
>> both a lat and lon).
>
> Maybe, there are tradeoffs though.
>
> Let's get concrete and look at the VectorDistanceFunction (dist()). It can currently take in an even number of ValueSource instances, and the distance method essentially boils down to (for Euclidean Distance):
> for (int i = 0; i < docValues1.length; i++) {
> double v = docValues1[i].doubleVal(doc) - docValues2[i].doubleVal(doc);
> result += v * v;
> }
> result = Math.sqrt(result);
>
> For example, a call to this might be:
> dist(power = 2, x1, y1, x2, y2) - where xi, yi are ValueSources. //note power = 2 is just me showing what the first parameter is so that no one wonders why there is this extra number in there
>
> Now, assuming a PointType fields named point1 and point 2 (along with the others above), one could have:
>
> dist(2, point1, point2) //distance between two PointTypes
> dist(2, point1, x1, y2) //distance between a PointType and a user defined point.
>
D'oh, I see another way of doing this, namely the Distance functions only work with points.
Namely, the second case above becomes:
dist(2, point1, point(x1, y1));
-Grant
> While I think this can be coded up in the lat/lon case (i.e. two values) I think it gets hairy when you consider a point in n-dim. space.
>
> My inclination is to fudge on this and do something in ValueSourceParser for each of the functions that can deal w/ poly fields (my gut says most can't) like:
> addParser("dist", new ValueSourceParser() {
> public ValueSource parse(FunctionQParser fp) throws ParseException {
> float power = fp.parseFloat();
> List<ValueSource> sources = fp.parseValueSourceList();
> if (sources.size() % 2 != 0) {
> //expand if needed
> List newSources = new ...;
> for each sources
> if (source is a PolyValueSource){
> List<ValueSource> tmp = ((PolyValueSource)source).getValueSources();
> newSources.addAll(tmp);
> else
> newSources.add(source); //just like the old one
> sources = newSources;
> //Do the even check again here
> if (sources.size() % 2 != 0){
> throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Illegal number of sources. There must be an even number of sources");
> }
> }
> int dim = sources.size() / 2;
> List<ValueSource> sources1 = new ArrayList<ValueSource>(dim);
> List<ValueSource> sources2 = new ArrayList<ValueSource>(dim);
> splitSources(dim, sources, sources1, sources2);
> return new VectorDistanceFunction(power, sources1, sources2);
> }
> });
>
> Of course, this requires documentation, etc. for others to be able to do the same for their custom Functions, but that is surmountable.
>
> -Grant
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene:
http://www.lucidimagination.com/search
Re: SOLR-1131: Poly Fields and ValueSource
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 10, 2009, at 1:26 PM, Yonik Seeley wrote:
> I wouldn't necessarily link FieldType.isPolyField() to the idea of a
> poly value source... they are two different things.
Yep. The word Poly is overloaded here to mean multiple ValueSources, but it isn't necessarily tied to there being a poly field, even though a PolyField likely would create a PolyValueSource
> For example, if NumericField had not already been written in Lucene, I
> would have perhaps just indexed both the lat and lon into the same
> lucene field. That part can be more of an implementation detail, and
> does not reflect the semantics of the field (the fact that it contains
> both a lat and lon).
Maybe, there are tradeoffs though.
Let's get concrete and look at the VectorDistanceFunction (dist()). It can currently take in an even number of ValueSource instances, and the distance method essentially boils down to (for Euclidean Distance):
for (int i = 0; i < docValues1.length; i++) {
double v = docValues1[i].doubleVal(doc) - docValues2[i].doubleVal(doc);
result += v * v;
}
result = Math.sqrt(result);
For example, a call to this might be:
dist(power = 2, x1, y1, x2, y2) - where xi, yi are ValueSources. //note power = 2 is just me showing what the first parameter is so that no one wonders why there is this extra number in there
Now, assuming a PointType fields named point1 and point 2 (along with the others above), one could have:
dist(2, point1, point2) //distance between two PointTypes
dist(2, point1, x1, y2) //distance between a PointType and a user defined point.
While I think this can be coded up in the lat/lon case (i.e. two values) I think it gets hairy when you consider a point in n-dim. space.
My inclination is to fudge on this and do something in ValueSourceParser for each of the functions that can deal w/ poly fields (my gut says most can't) like:
addParser("dist", new ValueSourceParser() {
public ValueSource parse(FunctionQParser fp) throws ParseException {
float power = fp.parseFloat();
List<ValueSource> sources = fp.parseValueSourceList();
if (sources.size() % 2 != 0) {
//expand if needed
List newSources = new ...;
for each sources
if (source is a PolyValueSource){
List<ValueSource> tmp = ((PolyValueSource)source).getValueSources();
newSources.addAll(tmp);
else
newSources.add(source); //just like the old one
sources = newSources;
//Do the even check again here
if (sources.size() % 2 != 0){
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Illegal number of sources. There must be an even number of sources");
}
}
int dim = sources.size() / 2;
List<ValueSource> sources1 = new ArrayList<ValueSource>(dim);
List<ValueSource> sources2 = new ArrayList<ValueSource>(dim);
splitSources(dim, sources, sources1, sources2);
return new VectorDistanceFunction(power, sources1, sources2);
}
});
Of course, this requires documentation, etc. for others to be able to do the same for their custom Functions, but that is surmountable.
-Grant
Re: SOLR-1131: Poly Fields and ValueSource
Posted by Yonik Seeley <yo...@lucidimagination.com>.
I wouldn't necessarily link FieldType.isPolyField() to the idea of a
poly value source... they are two different things.
For example, if NumericField had not already been written in Lucene, I
would have perhaps just indexed both the lat and lon into the same
lucene field. That part can be more of an implementation detail, and
does not reflect the semantics of the field (the fact that it contains
both a lat and lon).
-Yonik
http://www.lucidimagination.com
On Thu, Dec 10, 2009 at 1:16 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Dec 10, 2009, at 12:37 PM, Grant Ingersoll wrote:
>
>> OK, next up:
>>
>> How to handle the need to create multiple ValueSource instances for a given poly field? FieldType.getValueSource() only returns a single instance.
>>
>> I think there are a few options:
>>
>> 1. Change the signature above to return a list of ValueSources. This likely has back compat. issues.
>> 2. Just change those functions that need to handle poly fields (the distance functions) to worry about them and realize that the other functions won't work with them
>> 3. Punt and require the user to know the secret handshake that is the naming convention used. In some regard, this is no different than what any developer would experience now when using dynamic fields with function queries.
>
> 4. Create a derivative ValueSource called PolyValueSource that is a ValueSource that can "expand" (kind of like a rewrite() method) to get the "sub" ValueSources. Then, the functions that can work with a poly field can check to see if the ValueSource is expandable and then do so. This way all existing functions work as is, we just need to make a couple of changes in the ValueSourceParser when dealing with the distance functions. This is a combination of 1 and 2, essentially.
>
> 5. Yonik suggested modifying DocValues (DocValues.polyVal(double [] valHolder) - Yonik called it getPoint(double [] val) but I think it should be more generic since a poly field isn't point specific ) to be able to return multiple values per Document.
> GSI: I'm not sure this works w/o then needing to add in checks in the functions themselves to deal w/ poly doc values, but I could be missing something.
Re: SOLR-1131: Poly Fields and ValueSource
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 10, 2009, at 12:37 PM, Grant Ingersoll wrote:
> OK, next up:
>
> How to handle the need to create multiple ValueSource instances for a given poly field? FieldType.getValueSource() only returns a single instance.
>
> I think there are a few options:
>
> 1. Change the signature above to return a list of ValueSources. This likely has back compat. issues.
> 2. Just change those functions that need to handle poly fields (the distance functions) to worry about them and realize that the other functions won't work with them
> 3. Punt and require the user to know the secret handshake that is the naming convention used. In some regard, this is no different than what any developer would experience now when using dynamic fields with function queries.
4. Create a derivative ValueSource called PolyValueSource that is a ValueSource that can "expand" (kind of like a rewrite() method) to get the "sub" ValueSources. Then, the functions that can work with a poly field can check to see if the ValueSource is expandable and then do so. This way all existing functions work as is, we just need to make a couple of changes in the ValueSourceParser when dealing with the distance functions. This is a combination of 1 and 2, essentially.
5. Yonik suggested modifying DocValues (DocValues.polyVal(double [] valHolder) - Yonik called it getPoint(double [] val) but I think it should be more generic since a poly field isn't point specific ) to be able to return multiple values per Document.
GSI: I'm not sure this works w/o then needing to add in checks in the functions themselves to deal w/ poly doc values, but I could be missing something.