You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Steven White <sw...@gmail.com> on 2021/02/14 16:00:42 UTC
SolrJ: SolrInputDocument.addField()
Hi everyone,
It looks like I'm misusing SolrJ API SolrInputDocument.addField() thus I
need clarification.
Here is an example of what I have in my code:
SolrInputDocument doc = new SolrInputDocument();
doc.addField("MyFieldOne", "some data");
doc.addField("MyFieldTwo", 100);
The above code is creating 2 fields for me (if they don't exist already)
and then indexing the data to those fields. The data is "some data" and
the number 100 However, when the field is created, it is not using the
field type that I custom created in my schema. My question is, how do I
tell addField() to use my custom field type?
I _think_ I have to first SolrInputDocument.createField() and then call
SolrInputDocument.addField()? Or is the process of indexing data into a
field done via some other API I overlooked?
I need some guidance to make sure I get the logic right.
Thanks.
Steven
Re: SolrJ: SolrInputDocument.addField()
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/15/2021 10:17 AM, Steven White wrote:
> Yes, I have managed schema enabled like so:
>
> <schemaFactory class="ManagedIndexSchemaFactory">
> <bool name="mutable">true</bool>
> <str name="managedSchemaResourceName">cp-schema.xml</str>
> </schemaFactory>
>
> The reason why I enabled it is so that I can dynamically customize the
> schema based on what's in the DB. So that I can add fields to the schema
> dynamically.
A managed/mutable schema is a configuration detail that's separate from
(and required by) the update processor that guesses unknown fields. It
has been the default schema factory used in out-of-the box
configurations for quite a while.
> I guess a better question, to meet my need, is this: how do I tell Solr, in
> schema-less mode, to use *my* defined field-type whenever it needs to
> create a new field?
The config for that is described here:
https://lucene.apache.org/solr/guide/8_6/schemaless-mode.html#enable-field-class-guessing
It is a bad idea to rely on field guessing for a production index. Even
the most carefully designed configuration cannot get it right every
time. You're very likely to run into situations where the software's
best guess turns out to be wrong for your needs. And then you're forced
into what you should have done in the first place -- manually fixing the
definition for that field, which usually also requires reindexing from
scratch.
One counter-argument to what I stated in the last paragraph that
frequently comes up is "my data is very well curated and consistent."
But if that is the case, then you will know what fields and types are
required *in advance* and you can easily construct a schema yourself
before sending any data for indexing -- no guessing required.
Thanks,
Shawn
Re: SolrJ: SolrInputDocument.addField()
Posted by Steven White <sw...@gmail.com>.
Hi Shawn,
Yes, I have managed schema enabled like so:
<schemaFactory class="ManagedIndexSchemaFactory">
<bool name="mutable">true</bool>
<str name="managedSchemaResourceName">cp-schema.xml</str>
</schemaFactory>
The reason why I enabled it is so that I can dynamically customize the
schema based on what's in the DB. So that I can add fields to the schema
dynamically.
I didn't know about the field "guessing" part. Now that I know I see this
in my solrconfig.xml file:
<updateRequestProcessorChain name="add-unknown-fields-to-the-schema"
default="${update.autoCreateFields:true}"
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
<processor class="solr.LogUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory"/>
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
If I remove this block, what will happen?
I guess a better question, to meet my need, is this: how do I tell Solr, in
schema-less mode, to use *my* defined field-type whenever it needs to
create a new field?
I'm on Solr 8.6.1 and the link at
https://lucene.apache.org/solr/guide/8_6/schema-factory-definition-in-solrconfig.html#schema-factory-definition-in-solrconfig
doesn't offer much help.
Thanks
Steven
On Mon, Feb 15, 2021 at 11:09 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/15/2021 6:52 AM, Steven White wrote:
> > It looks to me that SolrInputDocument.addField() is either missnamed or
> > isn't well implemented.
> >
> > When it is called on a field that doesn't exist in the schema, it will
> > create that field and give it a type based on the data. Not only that,
> it
> > will set default values. For example, this call
> >
> > SolrInputDocument doc = new SolrInputDocument();
> > doc.addField("Company", "ACM company");
> >
> > Will create the following:
> >
> > <field name="Company" type="text_general"/>
> > <copyField source="Company" dest="Company_str" maxChars="256"/>
>
> That SolrJ code does not make those changes to your schema. At least
> not in the way you're thinking.
>
> It sounds to me like your solrconfig.xml includes what we call
> "schemaless mode" -- an update processor that adds unknown fields when
> they are indexed. You should disable it. We strongly recommend never
> using it in production, because it can make the wrong guess about which
> fieldType is required. The fieldType chosen has very little to do with
> the SolrJ code. It is controlled by what's in solrconfig.xml.
>
> Thanks,
> Shawn
>
Re: SolrJ: SolrInputDocument.addField()
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/15/2021 6:52 AM, Steven White wrote:
> It looks to me that SolrInputDocument.addField() is either missnamed or
> isn't well implemented.
>
> When it is called on a field that doesn't exist in the schema, it will
> create that field and give it a type based on the data. Not only that, it
> will set default values. For example, this call
>
> SolrInputDocument doc = new SolrInputDocument();
> doc.addField("Company", "ACM company");
>
> Will create the following:
>
> <field name="Company" type="text_general"/>
> <copyField source="Company" dest="Company_str" maxChars="256"/>
That SolrJ code does not make those changes to your schema. At least
not in the way you're thinking.
It sounds to me like your solrconfig.xml includes what we call
"schemaless mode" -- an update processor that adds unknown fields when
they are indexed. You should disable it. We strongly recommend never
using it in production, because it can make the wrong guess about which
fieldType is required. The fieldType chosen has very little to do with
the SolrJ code. It is controlled by what's in solrconfig.xml.
Thanks,
Shawn
Re: SolrJ: SolrInputDocument.addField()
Posted by Steven White <sw...@gmail.com>.
Thanks Shawn.
It looks to me that SolrInputDocument.addField() is either missnamed or
isn't well implemented.
When it is called on a field that doesn't exist in the schema, it will
create that field and give it a type based on the data. Not only that, it
will set default values. For example, this call
SolrInputDocument doc = new SolrInputDocument();
doc.addField("Company", "ACM company");
Will create the following:
<field name="Company" type="text_general"/>
<copyField source="Company" dest="Company_str" maxChars="256"/>
Since this is happening without the caller knowing it (this is not
documented) it leads to search issues (the intended analyzer will not be
used to name one).
It looks to me that the only way I can fix this is to first create the
field type first and then call addField() passing it the field I
just created. To that end, I cannot find a SolrJ API to create a field
type. Is that the case?
Steven.
On Sun, Feb 14, 2021 at 7:17 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/14/2021 9:00 AM, Steven White wrote:
> > It looks like I'm misusing SolrJ API SolrInputDocument.addField() thus I
> > need clarification.
> >
> > Here is an example of what I have in my code:
> >
> > SolrInputDocument doc = new SolrInputDocument();
> > doc.addField("MyFieldOne", "some data");
> > doc.addField("MyFieldTwo", 100);
> >
> > The above code is creating 2 fields for me (if they don't exist already)
> > and then indexing the data to those fields. The data is "some data" and
> > the number 100 However, when the field is created, it is not using the
> > field type that I custom created in my schema. My question is, how do I
> > tell addField() to use my custom field type?
>
> There is no way in SolrJ code to control which fieldType is used. That
> is controlled solely by the server-side schema definition.
>
> How do you know that Solr is not using the correct fieldType? If you
> are looking at the documents returned by a search and aren't seeing the
> transformations described in the schema, you're looking in the wrong place.
>
> Solr search results always returns what was originally sent in for
> indexing. Only Update Processors (defined in solrconfig.xml, not the
> schema) can affect what gets returned in results, fieldType definitions
> NEVER affect data returned in search results.
>
> Thanks,
> Shawn
>
Re: SolrJ: SolrInputDocument.addField()
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/14/2021 9:00 AM, Steven White wrote:
> It looks like I'm misusing SolrJ API SolrInputDocument.addField() thus I
> need clarification.
>
> Here is an example of what I have in my code:
>
> SolrInputDocument doc = new SolrInputDocument();
> doc.addField("MyFieldOne", "some data");
> doc.addField("MyFieldTwo", 100);
>
> The above code is creating 2 fields for me (if they don't exist already)
> and then indexing the data to those fields. The data is "some data" and
> the number 100 However, when the field is created, it is not using the
> field type that I custom created in my schema. My question is, how do I
> tell addField() to use my custom field type?
There is no way in SolrJ code to control which fieldType is used. That
is controlled solely by the server-side schema definition.
How do you know that Solr is not using the correct fieldType? If you
are looking at the documents returned by a search and aren't seeing the
transformations described in the schema, you're looking in the wrong place.
Solr search results always returns what was originally sent in for
indexing. Only Update Processors (defined in solrconfig.xml, not the
schema) can affect what gets returned in results, fieldType definitions
NEVER affect data returned in search results.
Thanks,
Shawn