Keep some fields intact while re-indexing documents

[Yaşar Arabacı <ya...@gmail.com>]

Hi,

I have some documents (around 80K, grows by ~1K each week) in mongodb
that I import weekly into solr using a custom python script.  I want
to add new fields to documents that don't exist in mongodb (like
favorite, hidden, comments etc).

I made a small test and it looks like when I re-import data from
mongodb, my custom fields get deleted.

I am planning to merge mongodb fields and my custom fields in python
script before reindexing, but I was wondering if there is a
better/recommended way.

Best Regards,

Re: Keep some fields intact while re-indexing documents

[Dmitri Maziuk <dm...@gmail.com>]

On 11/27/25 04:51, Yaşar Arabacı wrote:
> Hi Ilan,
> 
> Thanks for the response. Is there a "update or create" type of query,
> or should I check if document exists before making changes?

Read up on the _version field under "optimistic concurrency".

Keep in mind that if you have copyfields in the mix, you likely want to 
be updating all on their sources, or none.

Dima

Re: Keep some fields intact while re-indexing documents

[Yaşar Arabacı <ya...@gmail.com>]

Hi Ilan,

Thanks for the response. Is there a "update or create" type of query,
or should I check if document exists before making changes?

Best Regards,

Ilan Ginzburg <il...@gmail.com>, 27 Kas 2025 Per, 13:44 tarihinde şunu yazdı:
>
> You might want to look at Partial Updates
> <https://solr.apache.org/guide/solr/latest/indexing-guide/partial-document-updates.html>
> .
> With this approach, when you reindex from the DB, you update in the index
> the fields that come from the DB and keep the values of the other fields
> unchanged. This is achieved by storing in the index the values of these
> other fields (in addition to indexing them) so that Solr can retrieve them
> then index them back with the update coming from the DB.
>
> Ilan
>
> On Thu, Nov 27, 2025 at 11:31 AM Yaşar Arabacı <ya...@gmail.com> wrote:
>
> > Hi,
> >
> > I have some documents (around 80K, grows by ~1K each week) in mongodb
> > that I import weekly into solr using a custom python script.  I want
> > to add new fields to documents that don't exist in mongodb (like
> > favorite, hidden, comments etc).
> >
> > I made a small test and it looks like when I re-import data from
> > mongodb, my custom fields get deleted.
> >
> > I am planning to merge mongodb fields and my custom fields in python
> > script before reindexing, but I was wondering if there is a
> > better/recommended way.
> >
> > Best Regards,
> >

Re: Keep some fields intact while re-indexing documents

[Ilan Ginzburg <il...@gmail.com>]

You might want to look at Partial Updates
<https://solr.apache.org/guide/solr/latest/indexing-guide/partial-document-updates.html>
.
With this approach, when you reindex from the DB, you update in the index
the fields that come from the DB and keep the values of the other fields
unchanged. This is achieved by storing in the index the values of these
other fields (in addition to indexing them) so that Solr can retrieve them
then index them back with the update coming from the DB.

Ilan

On Thu, Nov 27, 2025 at 11:31 AM Yaşar Arabacı <ya...@gmail.com> wrote:

> Hi,
>
> I have some documents (around 80K, grows by ~1K each week) in mongodb
> that I import weekly into solr using a custom python script.  I want
> to add new fields to documents that don't exist in mongodb (like
> favorite, hidden, comments etc).
>
> I made a small test and it looks like when I re-import data from
> mongodb, my custom fields get deleted.
>
> I am planning to merge mongodb fields and my custom fields in python
> script before reindexing, but I was wondering if there is a
> better/recommended way.
>
> Best Regards,
>