You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Steven White <sw...@gmail.com> on 2019/07/14 14:41:55 UTC
How to query against dynamic fields without listing them all?
Hi everyone,
In my schema, I have the following field:
<dynamicField name="CC_COMP_NAME_*" type="file_text" multiValued="true"
indexed="true" required="false" stored="false"/>
When I index, I create dynamic fields and index into it like so:
doc.addField("CC_COMP_NAME_" + componentName.toUpperCase(),
ccAllFieldsDataValue);
In my query handler, I have this:
{"requestHandler":{"/select_hcl":{
"class":"solr.SearchHandler",
"name":"/select_hcl",
"defaults":{
"defType":"edismax",
"echoParams":"explicit",
"fl":"CC_UNIQUE_FIELD,CC_FILE_PATH,score",
"indent":"true",
"qf":"CC_COMP_NAME_*",
"rows":"100",
"wt":"xml"}}}}
My expectation was when i query using this handler, it will include all the
dynamic fields with the prefix of "CC_COMP_NAME_" however, that is not
happening and I'm getting 0 hits. But when I use the full field name, such
as CC_COMP_NAME_1 or CC_COMP_NAME_2, that works so I know my data is
indexed, it is just that Solr not paying attention to the dynamic field
syntax in "qf".
I don't want to keep a list of those dynamic fields and pass them to my
handler, but if I must, than I must. If so, how can I get the list of
those dynamic fields from Solr so that I don't have to maintain and sync-up
the list myself.
Thanks
Steven
Re: How to query against dynamic fields without listing them all?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
The other options is to use query field alias:
https://lucene.apache.org/solr/guide/8_1/the-extended-dismax-query-parser.html#field-aliasing-using-per-field-qf-overrides
(example: https://github.com/arafalov/solr-indexing-book/blob/master/published/languages/conf/solrconfig.xml#L20
). This still does not allow wildcards (I think, check!), but at least
it allows to isolate field names as a reference in a separate
variable.
And you can update that variable using Request Parameter API:
https://lucene.apache.org/solr/guide/8_1/request-parameters-api.html#request-parameters-api
You can periodically get the list of all fields, using Luke endpoint.
I don't remember if/where it is documented, but you can examine the
traffic between Admin UI (Analysis or Schema screen) and backend to
see how it works.
You may even have a custom postCommit listener defined that could
check the current list of dynamic fields and write appropriate
mapping: https://lucene.apache.org/solr/guide/8_1/updatehandlers-in-solrconfig.html#event-listeners
Regards,
Alex.
P.s. If you use copyField and the target does not store content, just
index - then the overhead is not too bad. And if your original fields
are store-only (because they are never searched directly) and the
copyField target is index-only, then there is no extra overhead.
On Sun, 14 Jul 2019 at 16:37, Steven White <sw...@gmail.com> wrote:
>
> Thanks David. Yes, I see that would work, but wouldn't that increase my
> index size? It would be best if Solr supported dynamic field in "qf".
>
> Steven.
>
> On Sun, Jul 14, 2019 at 11:02 AM David Santamauro <
> david.santamauro@gmail.com> wrote:
>
> > Hi Steven,
> >
> > You can dump all the dynamic fields into a copyField
> >
> > <field name="CC_COMP_NAME_ALL" type="file _text" indexed="true"
> > stored="false" multiValued="true" />
> > <copyField source="CC_COMP_NAME_*" dest="CC_COMP_NAME_ALL" />
> >
> > Then you can just set
> > "qf":"CC_COMP_NAME_ALL"
> >
> >
> > On 7/14/19, 10:42 AM, "Steven White" <sw...@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > In my schema, I have the following field:
> >
> > <dynamicField name="CC_COMP_NAME_*" type="file_text"
> > multiValued="true"
> > indexed="true" required="false" stored="false"/>
> >
> > When I index, I create dynamic fields and index into it like so:
> >
> > doc.addField("CC_COMP_NAME_" + componentName.toUpperCase(),
> > ccAllFieldsDataValue);
> >
> > In my query handler, I have this:
> >
> > {"requestHandler":{"/select_hcl":{
> > "class":"solr.SearchHandler",
> > "name":"/select_hcl",
> > "defaults":{
> > "defType":"edismax",
> > "echoParams":"explicit",
> > "fl":"CC_UNIQUE_FIELD,CC_FILE_PATH,score",
> > "indent":"true",
> > "qf":"CC_COMP_NAME_*",
> > "rows":"100",
> > "wt":"xml"}}}}
> >
> > My expectation was when i query using this handler, it will include
> > all the
> > dynamic fields with the prefix of "CC_COMP_NAME_" however, that is not
> > happening and I'm getting 0 hits. But when I use the full field name,
> > such
> > as CC_COMP_NAME_1 or CC_COMP_NAME_2, that works so I know my data is
> > indexed, it is just that Solr not paying attention to the dynamic field
> > syntax in "qf".
> >
> > I don't want to keep a list of those dynamic fields and pass them to my
> > handler, but if I must, than I must. If so, how can I get the list of
> > those dynamic fields from Solr so that I don't have to maintain and
> > sync-up
> > the list myself.
> >
> > Thanks
> >
> > Steven
> >
> >
Re: How to query against dynamic fields without listing them all?
Posted by Steven White <sw...@gmail.com>.
Thanks David. Yes, I see that would work, but wouldn't that increase my
index size? It would be best if Solr supported dynamic field in "qf".
Steven.
On Sun, Jul 14, 2019 at 11:02 AM David Santamauro <
david.santamauro@gmail.com> wrote:
> Hi Steven,
>
> You can dump all the dynamic fields into a copyField
>
> <field name="CC_COMP_NAME_ALL" type="file _text" indexed="true"
> stored="false" multiValued="true" />
> <copyField source="CC_COMP_NAME_*" dest="CC_COMP_NAME_ALL" />
>
> Then you can just set
> "qf":"CC_COMP_NAME_ALL"
>
>
> On 7/14/19, 10:42 AM, "Steven White" <sw...@gmail.com> wrote:
>
> Hi everyone,
>
> In my schema, I have the following field:
>
> <dynamicField name="CC_COMP_NAME_*" type="file_text"
> multiValued="true"
> indexed="true" required="false" stored="false"/>
>
> When I index, I create dynamic fields and index into it like so:
>
> doc.addField("CC_COMP_NAME_" + componentName.toUpperCase(),
> ccAllFieldsDataValue);
>
> In my query handler, I have this:
>
> {"requestHandler":{"/select_hcl":{
> "class":"solr.SearchHandler",
> "name":"/select_hcl",
> "defaults":{
> "defType":"edismax",
> "echoParams":"explicit",
> "fl":"CC_UNIQUE_FIELD,CC_FILE_PATH,score",
> "indent":"true",
> "qf":"CC_COMP_NAME_*",
> "rows":"100",
> "wt":"xml"}}}}
>
> My expectation was when i query using this handler, it will include
> all the
> dynamic fields with the prefix of "CC_COMP_NAME_" however, that is not
> happening and I'm getting 0 hits. But when I use the full field name,
> such
> as CC_COMP_NAME_1 or CC_COMP_NAME_2, that works so I know my data is
> indexed, it is just that Solr not paying attention to the dynamic field
> syntax in "qf".
>
> I don't want to keep a list of those dynamic fields and pass them to my
> handler, but if I must, than I must. If so, how can I get the list of
> those dynamic fields from Solr so that I don't have to maintain and
> sync-up
> the list myself.
>
> Thanks
>
> Steven
>
>
Re: How to query against dynamic fields without listing them all?
Posted by David Santamauro <da...@gmail.com>.
Hi Steven,
You can dump all the dynamic fields into a copyField
<field name="CC_COMP_NAME_ALL" type="file _text" indexed="true" stored="false" multiValued="true" />
<copyField source="CC_COMP_NAME_*" dest="CC_COMP_NAME_ALL" />
Then you can just set
"qf":"CC_COMP_NAME_ALL"
On 7/14/19, 10:42 AM, "Steven White" <sw...@gmail.com> wrote:
Hi everyone,
In my schema, I have the following field:
<dynamicField name="CC_COMP_NAME_*" type="file_text" multiValued="true"
indexed="true" required="false" stored="false"/>
When I index, I create dynamic fields and index into it like so:
doc.addField("CC_COMP_NAME_" + componentName.toUpperCase(),
ccAllFieldsDataValue);
In my query handler, I have this:
{"requestHandler":{"/select_hcl":{
"class":"solr.SearchHandler",
"name":"/select_hcl",
"defaults":{
"defType":"edismax",
"echoParams":"explicit",
"fl":"CC_UNIQUE_FIELD,CC_FILE_PATH,score",
"indent":"true",
"qf":"CC_COMP_NAME_*",
"rows":"100",
"wt":"xml"}}}}
My expectation was when i query using this handler, it will include all the
dynamic fields with the prefix of "CC_COMP_NAME_" however, that is not
happening and I'm getting 0 hits. But when I use the full field name, such
as CC_COMP_NAME_1 or CC_COMP_NAME_2, that works so I know my data is
indexed, it is just that Solr not paying attention to the dynamic field
syntax in "qf".
I don't want to keep a list of those dynamic fields and pass them to my
handler, but if I must, than I must. If so, how can I get the list of
those dynamic fields from Solr so that I don't have to maintain and sync-up
the list myself.
Thanks
Steven