You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jennifer Coston <Je...@raytheon.com> on 2016/08/16 16:49:25 UTC
Creating a SolrJ Data Service to send JSON to Solr
Hello,
I am trying to write a data service using SolrJ that will allow me to
accept JSON through a REST API, create a Solr document ,and write it to
multiple different Solr cores (depending on the core name specified). The
problem I am running into is that each core is going to have a different
schema. My current code has the common fields between all the schemas in a
data POJO which I then walk and set the values specified in the JSON to the
Solr Document. However, I don’t want to create a different class for each
schema to process the JSON and convert it to a Solr Document. Is there a
way to process the extra JSON fields that are not common between the
schemas and add them to the Solr Document, without knowing what they are
ahead of time? Is there a way to convert JSON to a Solr Document without
having to use a POJO? An alternative I was looking into is to use the
SolrClient to get the schema fields, create a POJO, walk that POJO to
create a Solr Document and then add it to Solr but, it doesn’t seem to be
possible to obtain the fields this way.
I know that the easiest way to add JSON to Solr would be to use a curl
command and send the JSON directly to Solr but this doesn’t match our
requirements, so I need to figure out a way to perform the same operation
using SolrJ. Any other ideas or suggestions would be greatly appreciated!
Thank you,
-Jennifer
Re: Creating a SolrJ Data Service to send JSON to Solr
Posted by Jennifer Coston <Je...@raytheon.com>.
Thank you Alex and Anshum! I will look into both of these.
Jennifer
From: Anshum Gupta <an...@anshumgupta.net>
To: solr-user@lucene.apache.org
Date: 08/16/2016 08:17 PM
Subject: Re: Creating a SolrJ Data Service to send JSON to Solr
I would also suggest sending the JSON directly to the JSON end point, with
the mapping :
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index
+Handlers#UploadingDatawithIndexHandlers-JSONUpdateConveniencePaths
On Tue, Aug 16, 2016 at 4:43 PM Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> Why do you need a POJO? For Solr purposes, you could just get the
> field names from schema and use those to map directly from JSON to the
> 'addField' calls in SolrDocument.
>
> Do you need it for non-Solr purposes? Then you can search for generic
> Java dynamic POJO generation solution.
>
> Also, you could look at creating a superset rather than common-subset
> POJO and then ignore all unknown fields on Solr side by adding a
> dynamicField that matches '*' with everything (index, store,
> docValues) set to false.
>
> Regards,
> Alex.
>
> ----
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 17 August 2016 at 02:49, Jennifer Coston
> <Je...@raytheon.com> wrote:
> >
> > Hello,
> > I am trying to write a data service using SolrJ that will allow me to
> > accept JSON through a REST API, create a Solr document ,and write it to
> > multiple different Solr cores (depending on the core name specified).
The
> > problem I am running into is that each core is going to have a
different
> > schema. My current code has the common fields between all the schemas
in
> a
> > data POJO which I then walk and set the values specified in the JSON to
> the
> > Solr Document. However, I don’t want to create a different class for
each
> > schema to process the JSON and convert it to a Solr Document. Is there
a
> > way to process the extra JSON fields that are not common between the
> > schemas and add them to the Solr Document, without knowing what they
are
> > ahead of time? Is there a way to convert JSON to a Solr Document
without
> > having to use a POJO? An alternative I was looking into is to use the
> > SolrClient to get the schema fields, create a POJO, walk that POJO to
> > create a Solr Document and then add it to Solr but, it doesn’t seem to
be
> > possible to obtain the fields this way.
> >
> > I know that the easiest way to add JSON to Solr would be to use a curl
> > command and send the JSON directly to Solr but this doesn’t match our
> > requirements, so I need to figure out a way to perform the same
operation
> > using SolrJ. Any other ideas or suggestions would be greatly
appreciated!
> >
> > Thank you,
> >
> > -Jennifer
>
Re: Creating a SolrJ Data Service to send JSON to Solr
Posted by Anshum Gupta <an...@anshumgupta.net>.
I would also suggest sending the JSON directly to the JSON end point, with
the mapping :
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-JSONUpdateConveniencePaths
On Tue, Aug 16, 2016 at 4:43 PM Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> Why do you need a POJO? For Solr purposes, you could just get the
> field names from schema and use those to map directly from JSON to the
> 'addField' calls in SolrDocument.
>
> Do you need it for non-Solr purposes? Then you can search for generic
> Java dynamic POJO generation solution.
>
> Also, you could look at creating a superset rather than common-subset
> POJO and then ignore all unknown fields on Solr side by adding a
> dynamicField that matches '*' with everything (index, store,
> docValues) set to false.
>
> Regards,
> Alex.
>
> ----
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 17 August 2016 at 02:49, Jennifer Coston
> <Je...@raytheon.com> wrote:
> >
> > Hello,
> > I am trying to write a data service using SolrJ that will allow me to
> > accept JSON through a REST API, create a Solr document ,and write it to
> > multiple different Solr cores (depending on the core name specified). The
> > problem I am running into is that each core is going to have a different
> > schema. My current code has the common fields between all the schemas in
> a
> > data POJO which I then walk and set the values specified in the JSON to
> the
> > Solr Document. However, I don’t want to create a different class for each
> > schema to process the JSON and convert it to a Solr Document. Is there a
> > way to process the extra JSON fields that are not common between the
> > schemas and add them to the Solr Document, without knowing what they are
> > ahead of time? Is there a way to convert JSON to a Solr Document without
> > having to use a POJO? An alternative I was looking into is to use the
> > SolrClient to get the schema fields, create a POJO, walk that POJO to
> > create a Solr Document and then add it to Solr but, it doesn’t seem to be
> > possible to obtain the fields this way.
> >
> > I know that the easiest way to add JSON to Solr would be to use a curl
> > command and send the JSON directly to Solr but this doesn’t match our
> > requirements, so I need to figure out a way to perform the same operation
> > using SolrJ. Any other ideas or suggestions would be greatly appreciated!
> >
> > Thank you,
> >
> > -Jennifer
>
Re: Creating a SolrJ Data Service to send JSON to Solr
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Why do you need a POJO? For Solr purposes, you could just get the
field names from schema and use those to map directly from JSON to the
'addField' calls in SolrDocument.
Do you need it for non-Solr purposes? Then you can search for generic
Java dynamic POJO generation solution.
Also, you could look at creating a superset rather than common-subset
POJO and then ignore all unknown fields on Solr side by adding a
dynamicField that matches '*' with everything (index, store,
docValues) set to false.
Regards,
Alex.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 17 August 2016 at 02:49, Jennifer Coston
<Je...@raytheon.com> wrote:
>
> Hello,
> I am trying to write a data service using SolrJ that will allow me to
> accept JSON through a REST API, create a Solr document ,and write it to
> multiple different Solr cores (depending on the core name specified). The
> problem I am running into is that each core is going to have a different
> schema. My current code has the common fields between all the schemas in a
> data POJO which I then walk and set the values specified in the JSON to the
> Solr Document. However, I don’t want to create a different class for each
> schema to process the JSON and convert it to a Solr Document. Is there a
> way to process the extra JSON fields that are not common between the
> schemas and add them to the Solr Document, without knowing what they are
> ahead of time? Is there a way to convert JSON to a Solr Document without
> having to use a POJO? An alternative I was looking into is to use the
> SolrClient to get the schema fields, create a POJO, walk that POJO to
> create a Solr Document and then add it to Solr but, it doesn’t seem to be
> possible to obtain the fields this way.
>
> I know that the easiest way to add JSON to Solr would be to use a curl
> command and send the JSON directly to Solr but this doesn’t match our
> requirements, so I need to figure out a way to perform the same operation
> using SolrJ. Any other ideas or suggestions would be greatly appreciated!
>
> Thank you,
>
> -Jennifer