You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by mw...@iu.edu on 2025/02/18 17:19:11 UTC

More information about copyField?

So *how* does copyField work?  Do I wind up with two identical copies
of the data stored in the index (if stored='true')?  Does the
copyField element accept a 'stored' attribute, and can it have a
different value than the source field declares?  What *are* the
defined attributes for copyField, and do they interact with the source
field's attributes?  Where should I have found answers to these
questions?

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
library.indianapolis.iu.edu

Re: More information about copyField?

Posted by Rahul Goswami <ra...@gmail.com>.
The official documentation should be a good starting point for your
questions   :
https://solr.apache.org/guide/solr/latest/indexing-guide/copy-fields.html

Reading the above documentation should help answer some of them and provide
a general structure to your understanding of copyFields.

Overall, copyFields are not much different from a regular field in terms of
attributes that you can define (stored, indexed, docValues etc). It is a
destination for one or more of your source fields in cases when you might
want a catchall field, or a field with a different analysis pipeline for
search than the one defined for the source field.

What Solr does is before handing over the document for indexing to the
lower Lucene layer, based on the copy field configuration in your defined
schema, extracts fields from the document and copies them over to the copy
field to form the "complete" document.

From this point on, the copy field gets stored(if stored=true), analyzed
and indexed (if indexed=true) just like any other field.

-Rahul

On Tue, Feb 18, 2025 at 12:20 PM <mw...@iu.edu> wrote:

> So *how* does copyField work?  Do I wind up with two identical copies
> of the data stored in the index (if stored='true')?  Does the
> copyField element accept a 'stored' attribute, and can it have a
> different value than the source field declares?  What *are* the
> defined attributes for copyField, and do they interact with the source
> field's attributes?  Where should I have found answers to these
> questions?
>
> --
> Mark H. Wood
> Lead Technology Analyst
>
> University Library
> Indiana University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> library.indianapolis.iu.edu
>

Re: More information about copyField?

Posted by Walter Underwood <wu...@wunderwood.org>.
copyField is pretty straightforward. Before any field processing, it copies the data destined for one field to another field. The copyField element/directive doesn’t take any attributes. It copies, period. The destination field has stored, indexed, etc. The source field and destination field are completely independent and don’t “know” that the text came from a copy.

The most common use for this is to send the same text to one field processed as text for searching and another as a string field for faceting. Another use is a catch-all field that collects all the text.

Did the name change from IUPUI? I went to North Central High School.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 18, 2025, at 9:19 AM, mwood@iu.edu wrote:
> 
> So *how* does copyField work?  Do I wind up with two identical copies
> of the data stored in the index (if stored='true')?  Does the
> copyField element accept a 'stored' attribute, and can it have a
> different value than the source field declares?  What *are* the
> defined attributes for copyField, and do they interact with the source
> field's attributes?  Where should I have found answers to these
> questions?
> 
> -- 
> Mark H. Wood
> Lead Technology Analyst
> 
> University Library
> Indiana University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> library.indianapolis.iu.edu


Re: More information about copyField?

Posted by Walter Underwood <wu...@wunderwood.org>.
copyField is pretty straightforward. Before any field processing, it copies the data destined for one field to another field. The copyField element/directive doesn’t take any attributes. It copies, period. The destination field has stored, indexed, etc. The source field and destination field are completely independent and don’t “know” that the text came from a copy.

The most common use for this is to send the same text to one field processed as text for searching and another as a string field for faceting. Another use is a catch-all field that collects all the text.

Did the name change from IUPUI? I went to North Central High School.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 18, 2025, at 9:19 AM, mwood@iu.edu wrote:
> 
> So *how* does copyField work?  Do I wind up with two identical copies
> of the data stored in the index (if stored='true')?  Does the
> copyField element accept a 'stored' attribute, and can it have a
> different value than the source field declares?  What *are* the
> defined attributes for copyField, and do they interact with the source
> field's attributes?  Where should I have found answers to these
> questions?
> 
> -- 
> Mark H. Wood
> Lead Technology Analyst
> 
> University Library
> Indiana University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> library.indianapolis.iu.edu


Re: More information about copyField?

Posted by Walter Underwood <wu...@wunderwood.org>.
> On Feb 18, 2025, at 10:08 AM, mwood@iu.edu wrote:
> 
> Thank you.  This is good to know.  In
>  https://solr.apache.org/guide/solr/latest/indexing-guide/copy-fields.html
> there is no list of defined attributes and no statement that copyField
> accepts the same attributes as field.  The only attributes documented
> there are source, dest, and maxChars.

copyField is not a field. It is an instruction to duplicate the text heading to one field and also send it to another field.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


Re: More information about copyField?

Posted by Walter Underwood <wu...@wunderwood.org>.
> On Feb 18, 2025, at 10:08 AM, mwood@iu.edu wrote:
> 
> Thank you.  This is good to know.  In
>  https://solr.apache.org/guide/solr/latest/indexing-guide/copy-fields.html
> there is no list of defined attributes and no statement that copyField
> accepts the same attributes as field.  The only attributes documented
> there are source, dest, and maxChars.

copyField is not a field. It is an instruction to duplicate the text heading to one field and also send it to another field.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


Re: More information about copyField?

Posted by mw...@iu.edu.
On Tue, Feb 18, 2025 at 06:41:16PM +0000, Walter Underwood wrote:
> > On Feb 18, 2025, at 10:08 AM, mwood@iu.edu wrote:
> > 
> > Thank you.  This is good to know.  In
> >  https://solr.apache.org/guide/solr/latest/indexing-guide/copy-fields.html
> > there is no list of defined attributes and no statement that copyField
> > accepts the same attributes as field.  The only attributes documented
> > there are source, dest, and maxChars.
> 
> copyField is not a field. It is an instruction to duplicate the text heading to one field and also send it to another field.

This was the final missing piece.  That page is confusing, starting
with the title:  "Copy Fields".  The examples are all one-liners --
the context of copyField is never shown or stated.  If you will, read
that page and see if it doesn't make good sense on the assumption
that copyField declares a new field, not a relationship between
already-declared fields.

Today I found https://stackoverflow.com/q/4596075/2916377 which gives
enough context to see how copyField is used.

I appreciate everybody's patience.



Yes, IU and Purdue are splitting up in Indy.  We've become IU
Indianapolis.  Purdue is building its own Indianapolis campus next door
and elsewhere across the city.

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
library.indianapolis.iu.edu

Re: More information about copyField?

Posted by mw...@iu.edu.
On Tue, Feb 18, 2025 at 05:44:23PM +0000, Rahul Goswami wrote:
> The official documentation should be a good starting point for your
> questions   :
> https://solr.apache.org/guide/solr/latest/indexing-guide/copy-fields.html
> 
> Reading the above documentation should help answer some of them and provide
> a general structure to your understanding of copyFields.

Yes, it should have.  I had read it.

> Overall, copyFields are not much different from a regular field in terms of
> attributes that you can define (stored, indexed, docValues etc). It is a

Thank you.  This is good to know.  In
  https://solr.apache.org/guide/solr/latest/indexing-guide/copy-fields.html
there is no list of defined attributes and no statement that copyField
accepts the same attributes as field.  The only attributes documented
there are source, dest, and maxChars.

> destination for one or more of your source fields in cases when you might
> want a catchall field, or a field with a different analysis pipeline for
> search than the one defined for the source field.
> 
> What Solr does is before handing over the document for indexing to the
> lower Lucene layer, based on the copy field configuration in your defined
> schema, extracts fields from the document and copies them over to the copy
> field to form the "complete" document.

It's not stated in the page linked above whether the field value is
copied or referenced, and I've found no way to infer it.

> From this point on, the copy field gets stored(if stored=true), analyzed
> and indexed (if indexed=true) just like any other field.

Again, good to know.  Thank you.

This answers my immediate concern:  yes, we should pay attention to
the attributes of the source field and the copyField so that we can
avoid un-necessary duplication of storage.  We're working with full
texts of academic papers, which can be quite large, and we might save
quite a bit of storage and I/O with careful schema design.

> On Tue, Feb 18, 2025 at 12:20 PM <mw...@iu.edu> wrote:
> 
> > So *how* does copyField work?  Do I wind up with two identical copies
> > of the data stored in the index (if stored='true')?  Does the
> > copyField element accept a 'stored' attribute, and can it have a
> > different value than the source field declares?  What *are* the
> > defined attributes for copyField, and do they interact with the source
> > field's attributes?  Where should I have found answers to these
> > questions?
> >
> > --
> > Mark H. Wood
> > Lead Technology Analyst
> >
> > University Library
> > Indiana University Indianapolis
> > 755 W. Michigan Street
> > Indianapolis, IN 46202
> > 317-274-0749
> > library.indianapolis.iu.edu
> >

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
library.indianapolis.iu.edu