You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by JT <ha...@gmail.com> on 2013/10/10 21:56:14 UTC
Questions developing custom functionquery
I'm running into some issues developing a custom functionquery.
My goal is to be able to implement a custom sorting technique.
I have a field defined called resname, it is a single value str.
Example: <str name="resname">/some
example/data/here/2013/09/12/testing.text</str>
I would like to do a custom sort based on this resname field.
Basically, I would like to parse out that date there (2013/09/12) and sort
on that date.
I've followed various tutorials
- http://java.dzone.com/news/how-write-custom-solr
-
http://www.supermind.org/blog/756/how-to-write-a-custom-solr-functionquery
Im at the point where my code compiles, runs, executes, etc. Solr is happy
with my code.
I have classes that inherit from ValueSorceParser and ValueSorce, etc. I've
overrode parse and
instantiated my class with ValueSource
public ValueSource parse(FunctionQParser fqp) {
return MyCustomClass(fqp.parseValueSource)
}
public class MyCustomClass extends ValueSource {
ValueSource source;
public MyCustomClass(ValueSource source) {
this.source = source;
}
public FunctionValues getValues(....) {
final FunctionValues sourceDV =
source.getvalues(context,readerContext)
return new IntValues(this)
public int intVal(int doc) {
//parse the value of "resname" here
String value = sourceDV.strVal(doc);
...more stuff
}
}
}
The issue I'm running into is that my call to sourceDV.strVal(doc) only
returns "part" of the field, not all of it. It appears to be very random.
I guess my actual question is, how do I access / reference the EXACT RAW
value of a field, while writing a functionquery.
Do I need to change my ValueSource to a String?, then somehow lookup the
field name while inside my getValues call?
Is there a way to access the raw field data , when referencing it as a
FunctionValues?
Maybe I'm going about this totally incorrectly?
Re: Questions developing custom functionquery
Posted by JT <ha...@gmail.com>.
I would agree the "right" way to do this is probably just add the
information I wish to sort on directly, as a date field or something like
that.
The issue is we currently have ~300m documents that are already indexed.
Not all of the fields have stored=true (for good reason, we maintain the
documents externally, about 7TB worth. I didn't want to replicate 7TB of
data twice.) so we cannot update these indexed values.
I was hoping to spend 2-3 days writing a custom query to avoid 2+ months of
indexing everything all over again.
So let me just ask this question, given my current situation, lets say you
had the following field
<str name="resourcename">/path/to/file/month/day/year/file.txt</str>
I simply want to extract the month/day/year and sort based on that.
My current plan was to convert the month, day, year into seconds from right
now, and return that number. Thus sorting ascending, it should return
newest documents first.
-JT
On Fri, Oct 18, 2013 at 3:14 PM, Chris Hostetter
<ho...@fucit.org>wrote:
>
> : Field-Type: org.apache.solr.schema.TextField
> ...
> : DocTermsIndexDocValues<
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.lucene/lucene-queries/4.3.0/org/apache/lucene/queries/function/docvalues/DocTermsIndexDocValues.java#DocTermsIndexDocValues
> >.
> : Calling "getVal()" on a DocTermsIndexDocValues does some really weird
> stuff
> : that I really don't understand.
>
> Your TextField is being analyzed in some way you haven't clarified, and
> the DocTermsIndexDocValues you get contains the details of each term in
> that TextField
>
> : Its possible I'm going about this wrong and need to re-do my approach.
> I'm
> : just currently at a loss for what that approach is.
>
> Based on your initial goal, you are most certainly going about this in a
> much more complicated way then you need to...
>
> : > > > My goal is to be able to implement a custom sorting technique.
>
> : > > > Example: <str name="resname">/some
> : > > > example/data/here/2013/09/12/testing.text</str>
> : > > >
> : > > > I would like to do a custom sort based on this resname field.
> : > > > Basically, I would like to parse out that date there (2013/09/12)
> and
> : > > sort
> : > > > on that date.
>
> You are going to be *MUCH* happier (both in terms of effort, and in terms
> of performance) if instead of writing a custom function to parse strings
> at query time when sorting, you implement the parsing logic when indexing
> the doc and index it up front as a date field that you can sort on.
>
> I would suggest something like CloneFieldUpdateProcessorFactory +
> RegexReplaceProcessorFactory could save you the work of needing to
> implement any custom logic -- but as Jack pointed out in SOLR-4864 it
> doesn't currently allow you to do capture group replacements (but maybe
> you could contribute a patch to fix that instead of needing to write
> completely custom code for yourself)
>
> Of maybe, as is, you could use RegexReplaceProcessorFactory to throw away
> non digits - and then use ParseDateFieldUpdateProcessorFactory to get what
> you want? (I'm not certain - i haven't played with
> ParseDateFieldUpdateProcessorFactory much)
>
> https://issues.apache.org/jira/browse/SOLR-4864
>
> https://lucene.apache.org/solr/4_5_0/solr-core/org/apache/solr/update/processor/RegexReplaceProcessorFactory.html
>
> https://lucene.apache.org/solr/4_5_0/solr-core/org/apache/solr/update/processor/CloneFieldUpdateProcessorFactory.html
>
> https://lucene.apache.org/solr/4_5_0/solr-core/org/apache/solr/update/processor/ParseDateFieldUpdateProcessorFactory.html
>
>
>
> -Hoss
>
Re: Questions developing custom functionquery
Posted by Chris Hostetter <ho...@fucit.org>.
: Field-Type: org.apache.solr.schema.TextField
...
: DocTermsIndexDocValues<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.lucene/lucene-queries/4.3.0/org/apache/lucene/queries/function/docvalues/DocTermsIndexDocValues.java#DocTermsIndexDocValues>.
: Calling "getVal()" on a DocTermsIndexDocValues does some really weird stuff
: that I really don't understand.
Your TextField is being analyzed in some way you haven't clarified, and
the DocTermsIndexDocValues you get contains the details of each term in
that TextField
: Its possible I'm going about this wrong and need to re-do my approach. I'm
: just currently at a loss for what that approach is.
Based on your initial goal, you are most certainly going about this in a
much more complicated way then you need to...
: > > > My goal is to be able to implement a custom sorting technique.
: > > > Example: <str name="resname">/some
: > > > example/data/here/2013/09/12/testing.text</str>
: > > >
: > > > I would like to do a custom sort based on this resname field.
: > > > Basically, I would like to parse out that date there (2013/09/12) and
: > > sort
: > > > on that date.
You are going to be *MUCH* happier (both in terms of effort, and in terms
of performance) if instead of writing a custom function to parse strings
at query time when sorting, you implement the parsing logic when indexing
the doc and index it up front as a date field that you can sort on.
I would suggest something like CloneFieldUpdateProcessorFactory +
RegexReplaceProcessorFactory could save you the work of needing to
implement any custom logic -- but as Jack pointed out in SOLR-4864 it
doesn't currently allow you to do capture group replacements (but maybe
you could contribute a patch to fix that instead of needing to write
completely custom code for yourself)
Of maybe, as is, you could use RegexReplaceProcessorFactory to throw away
non digits - and then use ParseDateFieldUpdateProcessorFactory to get what
you want? (I'm not certain - i haven't played with
ParseDateFieldUpdateProcessorFactory much)
https://issues.apache.org/jira/browse/SOLR-4864
https://lucene.apache.org/solr/4_5_0/solr-core/org/apache/solr/update/processor/RegexReplaceProcessorFactory.html
https://lucene.apache.org/solr/4_5_0/solr-core/org/apache/solr/update/processor/CloneFieldUpdateProcessorFactory.html
https://lucene.apache.org/solr/4_5_0/solr-core/org/apache/solr/update/processor/ParseDateFieldUpdateProcessorFactory.html
-Hoss
Re: Questions developing custom functionquery
Posted by JT <ha...@gmail.com>.
Hey Mikhail,
Thanks for responding.
Field: resourcename
Field-Type: org.apache.solr.schema.TextField
All 9 boxes checked (indexed, tokenized, stored).
I have various other fields (including MD5-checksums) in my Schema. When I
use a md5sum field (which is a <str> field, but doesn't have spaces,
forward slashes, etc.) The plugin I've written performs exactly as I've
expected.
I think the large part of my problem is that my "ValueSource" is being
instantianted as a the class "StrFieldSource". When you call getvalues on a
"StrFieldSource", you end up with a
DocTermsIndexDocValues<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.lucene/lucene-queries/4.3.0/org/apache/lucene/queries/function/docvalues/DocTermsIndexDocValues.java#DocTermsIndexDocValues>.
Calling "getVal()" on a DocTermsIndexDocValues does some really weird stuff
that I really don't understand.
I assumed that calling ValueSource.getValues(...).strVal(int doc) would
simply return the "data" that my field corresponds to, but I don't think
that is true.
Its possible I'm going about this wrong and need to re-do my approach. I'm
just currently at a loss for what that approach is.
On Fri, Oct 11, 2013 at 2:48 AM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:
> Hello JT,
>
> what's is the field and fieldType definition for "resname" ?
> can't you check how '/some
> example/data/here/2013/09/12/
> testing.text
> ' is handled on analysis page in SolrAdmin?
>
>
> On Fri, Oct 11, 2013 at 4:53 AM, Richard Lee <ro...@gmail.com> wrote:
>
> > seems what u got is the terms other than the raw data. maybe u should
> check
> > the api docs for more details
> > 2013-10-11 上午3:56于 "JT" <ha...@gmail.com>写道：
> >
> > > I'm running into some issues developing a custom functionquery.
> > >
> > > My goal is to be able to implement a custom sorting technique.
> > >
> > > I have a field defined called resname, it is a single value str.
> > >
> > > Example: <str name="resname">/some
> > > example/data/here/2013/09/12/testing.text</str>
> > >
> > > I would like to do a custom sort based on this resname field.
> > > Basically, I would like to parse out that date there (2013/09/12) and
> > sort
> > > on that date.
> > >
> > >
> > > I've followed various tutorials
> > > - http://java.dzone.com/news/how-write-custom-solr
> > > -
> > >
> >
> http://www.supermind.org/blog/756/how-to-write-a-custom-solr-functionquery
> > >
> > >
> > > Im at the point where my code compiles, runs, executes, etc. Solr is
> > happy
> > > with my code.
> > >
> > > I have classes that inherit from ValueSorceParser and ValueSorce, etc.
> > I've
> > > overrode parse and
> > > instantiated my class with ValueSource
> > >
> > > public ValueSource parse(FunctionQParser fqp) {
> > > return MyCustomClass(fqp.parseValueSource)
> > > }
> > >
> > > public class MyCustomClass extends ValueSource {
> > > ValueSource source;
> > >
> > > public MyCustomClass(ValueSource source) {
> > > this.source = source;
> > > }
> > >
> > > public FunctionValues getValues(....) {
> > > final FunctionValues sourceDV =
> > > source.getvalues(context,readerContext)
> > > return new IntValues(this)
> > > public int intVal(int doc) {
> > > //parse the value of "resname" here
> > > String value = sourceDV.strVal(doc);
> > > ...more stuff
> > > }
> > > }
> > > }
> > >
> > > The issue I'm running into is that my call to sourceDV.strVal(doc) only
> > > returns "part" of the field, not all of it. It appears to be very
> random.
> > >
> > > I guess my actual question is, how do I access / reference the EXACT
> RAW
> > > value of a field, while writing a functionquery.
> > >
> > > Do I need to change my ValueSource to a String?, then somehow lookup
> the
> > > field name while inside my getValues call?
> > >
> > > Is there a way to access the raw field data , when referencing it as a
> > > FunctionValues?
> > >
> > >
> > > Maybe I'm going about this totally incorrectly?
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
Re: Questions developing custom functionquery
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello JT,
what's is the field and fieldType definition for "resname" ?
can't you check how '/some
example/data/here/2013/09/12/
testing.text
' is handled on analysis page in SolrAdmin?
On Fri, Oct 11, 2013 at 4:53 AM, Richard Lee <ro...@gmail.com> wrote:
> seems what u got is the terms other than the raw data. maybe u should check
> the api docs for more details
> 2013-10-11 上午3:56于 "JT" <ha...@gmail.com>写道：
>
> > I'm running into some issues developing a custom functionquery.
> >
> > My goal is to be able to implement a custom sorting technique.
> >
> > I have a field defined called resname, it is a single value str.
> >
> > Example: <str name="resname">/some
> > example/data/here/2013/09/12/testing.text</str>
> >
> > I would like to do a custom sort based on this resname field.
> > Basically, I would like to parse out that date there (2013/09/12) and
> sort
> > on that date.
> >
> >
> > I've followed various tutorials
> > - http://java.dzone.com/news/how-write-custom-solr
> > -
> >
> http://www.supermind.org/blog/756/how-to-write-a-custom-solr-functionquery
> >
> >
> > Im at the point where my code compiles, runs, executes, etc. Solr is
> happy
> > with my code.
> >
> > I have classes that inherit from ValueSorceParser and ValueSorce, etc.
> I've
> > overrode parse and
> > instantiated my class with ValueSource
> >
> > public ValueSource parse(FunctionQParser fqp) {
> > return MyCustomClass(fqp.parseValueSource)
> > }
> >
> > public class MyCustomClass extends ValueSource {
> > ValueSource source;
> >
> > public MyCustomClass(ValueSource source) {
> > this.source = source;
> > }
> >
> > public FunctionValues getValues(....) {
> > final FunctionValues sourceDV =
> > source.getvalues(context,readerContext)
> > return new IntValues(this)
> > public int intVal(int doc) {
> > //parse the value of "resname" here
> > String value = sourceDV.strVal(doc);
> > ...more stuff
> > }
> > }
> > }
> >
> > The issue I'm running into is that my call to sourceDV.strVal(doc) only
> > returns "part" of the field, not all of it. It appears to be very random.
> >
> > I guess my actual question is, how do I access / reference the EXACT RAW
> > value of a field, while writing a functionquery.
> >
> > Do I need to change my ValueSource to a String?, then somehow lookup the
> > field name while inside my getValues call?
> >
> > Is there a way to access the raw field data , when referencing it as a
> > FunctionValues?
> >
> >
> > Maybe I'm going about this totally incorrectly?
> >
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: Questions developing custom functionquery
Posted by Richard Lee <ro...@gmail.com>.
seems what u got is the terms other than the raw data. maybe u should check
the api docs for more details
2013-10-11 上午3:56于 "JT" <ha...@gmail.com>写道：
> I'm running into some issues developing a custom functionquery.
>
> My goal is to be able to implement a custom sorting technique.
>
> I have a field defined called resname, it is a single value str.
>
> Example: <str name="resname">/some
> example/data/here/2013/09/12/testing.text</str>
>
> I would like to do a custom sort based on this resname field.
> Basically, I would like to parse out that date there (2013/09/12) and sort
> on that date.
>
>
> I've followed various tutorials
> - http://java.dzone.com/news/how-write-custom-solr
> -
> http://www.supermind.org/blog/756/how-to-write-a-custom-solr-functionquery
>
>
> Im at the point where my code compiles, runs, executes, etc. Solr is happy
> with my code.
>
> I have classes that inherit from ValueSorceParser and ValueSorce, etc. I've
> overrode parse and
> instantiated my class with ValueSource
>
> public ValueSource parse(FunctionQParser fqp) {
> return MyCustomClass(fqp.parseValueSource)
> }
>
> public class MyCustomClass extends ValueSource {
> ValueSource source;
>
> public MyCustomClass(ValueSource source) {
> this.source = source;
> }
>
> public FunctionValues getValues(....) {
> final FunctionValues sourceDV =
> source.getvalues(context,readerContext)
> return new IntValues(this)
> public int intVal(int doc) {
> //parse the value of "resname" here
> String value = sourceDV.strVal(doc);
> ...more stuff
> }
> }
> }
>
> The issue I'm running into is that my call to sourceDV.strVal(doc) only
> returns "part" of the field, not all of it. It appears to be very random.
>
> I guess my actual question is, how do I access / reference the EXACT RAW
> value of a field, while writing a functionquery.
>
> Do I need to change my ValueSource to a String?, then somehow lookup the
> field name while inside my getValues call?
>
> Is there a way to access the raw field data , when referencing it as a
> FunctionValues?
>
>
> Maybe I'm going about this totally incorrectly?
>