You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Markus Jelsma <ma...@openindex.io> on 2018/03/29 21:06:26 UTC
PreAnalyzed FieldType, and simultaneously importing JSON
Hello,
We want to move to PreAnalyzed FieldType to offload our very heavy analysis chain away from the search cluster, so we have to configure our fields to accept pre-analyzed tokens in production.
But we use the same schema in development environments too, and that is where we use JSON files, or stream (export/import) data directly from production servers into a development environment, again via JSON. And in case of disaster recovery, we can import the daily exported JSON bzipped files back into our production servers.
But this JSON loading does not work with PreAnalyzed FieldType. So to load JSON we must reset all fields back to their respective language specific FieldTypes on-the-fly, we could automate, but it is a hassle we like to avoid.
Have i overlooked any configuration parameters that can help? Must we automate the on-the-fly schema reconfiguration and reset to PreAnalyzed after JSON loading is finished?
Many thanks!
Markus
RE: PreAnalyzed FieldType, and simultaneously importing JSON
Posted by Markus Jelsma <ma...@openindex.io>.
Hi David!
Many thanks, this looks much better!
Regards,
Markus
-----Original message-----
> From:David Smiley <da...@gmail.com>
> Sent: Monday 2nd April 2018 21:27
> To: solr-user@lucene.apache.org
> Subject: Re: PreAnalyzed FieldType, and simultaneously importing JSON
>
> Hello Markus,
>
> It appears you are not familiar with PreAnalyzedUpdateProcessor? Using
> that is much more flexible -- you could have different URP chains for your
> use-cases. IMO PreAnalyzedField ought to go away. I argued for the URP
> version and thus it's superiority to the FieldType here:
> https://issues.apache.org/jira/browse/SOLR-4619?focusedCommentId=13611191&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611191
> Sadly, the FieldType is the one that is documented in the ref guide, but
> not the URP :-(
>
> ~ David
>
> On Thu, Mar 29, 2018 at 5:06 PM Markus Jelsma <ma...@openindex.io>
> wrote:
>
> > Hello,
> >
> > We want to move to PreAnalyzed FieldType to offload our very heavy
> > analysis chain away from the search cluster, so we have to configure our
> > fields to accept pre-analyzed tokens in production.
> >
> > But we use the same schema in development environments too, and that is
> > where we use JSON files, or stream (export/import) data directly from
> > production servers into a development environment, again via JSON. And in
> > case of disaster recovery, we can import the daily exported JSON bzipped
> > files back into our production servers.
> >
> > But this JSON loading does not work with PreAnalyzed FieldType. So to load
> > JSON we must reset all fields back to their respective language specific
> > FieldTypes on-the-fly, we could automate, but it is a hassle we like to
> > avoid.
> >
> > Have i overlooked any configuration parameters that can help? Must we
> > automate the on-the-fly schema reconfiguration and reset to PreAnalyzed
> > after JSON loading is finished?
> >
> > Many thanks!
> > Markus
> >
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
>
Re: PreAnalyzed FieldType, and simultaneously importing JSON
Posted by David Smiley <da...@gmail.com>.
Hello Markus,
It appears you are not familiar with PreAnalyzedUpdateProcessor? Using
that is much more flexible -- you could have different URP chains for your
use-cases. IMO PreAnalyzedField ought to go away. I argued for the URP
version and thus it's superiority to the FieldType here:
https://issues.apache.org/jira/browse/SOLR-4619?focusedCommentId=13611191&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611191
Sadly, the FieldType is the one that is documented in the ref guide, but
not the URP :-(
~ David
On Thu, Mar 29, 2018 at 5:06 PM Markus Jelsma <ma...@openindex.io>
wrote:
> Hello,
>
> We want to move to PreAnalyzed FieldType to offload our very heavy
> analysis chain away from the search cluster, so we have to configure our
> fields to accept pre-analyzed tokens in production.
>
> But we use the same schema in development environments too, and that is
> where we use JSON files, or stream (export/import) data directly from
> production servers into a development environment, again via JSON. And in
> case of disaster recovery, we can import the daily exported JSON bzipped
> files back into our production servers.
>
> But this JSON loading does not work with PreAnalyzed FieldType. So to load
> JSON we must reset all fields back to their respective language specific
> FieldTypes on-the-fly, we could automate, but it is a hassle we like to
> avoid.
>
> Have i overlooked any configuration parameters that can help? Must we
> automate the on-the-fly schema reconfiguration and reset to PreAnalyzed
> after JSON loading is finished?
>
> Many thanks!
> Markus
>
--
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com