You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Abhijit Pawar <ab...@gmail.com> on 2017/03/28 22:08:40 UTC

Fieldtype json supported in SOLR 5.4.0 or 5.4.1

Hello All,

I am working on a requirement to index field of type JSON (in mongoDB
collection) in SOLR 5.4.0.

I am using mongo-jdbc-dih which I found on GitHub :

https://github.com/hrishik/solr-mongodb-dih

However I could not find a fieldtype on Apache SOLR wiki page which would
support JSON datatype in mongoDB.

Can someone please recommend a way to include datatype / fieldtype in SOLR
schema to support or index JSON data field from mongoDB.
Thanks.

R​egards,

Abhijit​

Re: Fieldtype json supported in SOLR 5.4.0 or 5.4.1

Posted by Abhijit Pawar <ap...@ibizsoftinc.com>.
Hi Edwin,
​​

I had to replace <<IP Address>> value with <<HOSTNAME>> in <dataSource
name="mongod" type="JdbcDataSource" driver="com.mongodb.jdbc.MongoDriver"
url="mongodb://<<>IP-Address>:27017/<<DB-NAME>>"/>

in my data-source-config file and was able to index the data.

​However w​
ith IP address it is still not working.
In solr-config.xml I am not seeing anywhere to mention schema.xml or
managed-schema.xml.

​Meanwhile going back to my original question - what fieldtype to use for
JSON field in mongoDB ?

Currently for field "pricing" in my data-source-config file​
​
​ I am using String Array as a datatype :

​
data-source-config file​
​
​ :
​

​<field column="pricing" name="pricing"/>​

​managed-schema.xml (for SOLR 6.0) /schema.xml(for older version of SOLR)​ :

<field name="pricing" type="string" indexed="true" stored="true"
​ multiValued="true"​

​/>

However it just shows and empty array when data is indexed.Any idea how to
define field of type JSON ?​
​

Best Regards,
<http://www.ibizsoftinc.com/>
<http://www.ibizsoftinc.com/>
Abhijit Pawar
Office : +1 (469) 287 2005 x 110



Follow us on:





On Sat, Apr 1, 2017 at 10:12 AM, Zheng Lin Edwin Yeo <ed...@gmail.com>
wrote:

> Did you upgrade your solrconfig.xml to the Solr 6.0 version too?
> There are some difference in Solr 6.0 version which requires setting to
> determine whether to use managed-schema or classic schema (the physical
> schema.xml file)
>
> Regards,
> Edwin
>
> On 1 April 2017 at 01:27, Abhijit Pawar <ab...@gmail.com>
> wrote:
>
> > Hi Rick,
> >
> > I tried installing SOLR 6.0 since SOLR 6.0 has managed-schema and tried
> > index the data from mongoDB :
> >
> >
> > <dataConfig>
> > <dataSource name="mongod" type="JdbcDataSource"
> > driver="com.mongodb.jdbc.MongoDriver" url="mongodb://
> > ​<<>IP-Address>​
> > :27017/
> > ​<<DB-NAME>>
> > "/>
> > <document name="products">
> > <entity name="products"
> > dataSource="mongod"
> > transformer="TemplateTransformer,ProdsCatsFieldTransformer"
> > onError="continue"
> > pk="uuid"
> > query="SELECT
> > orgidStr,idStr,name,code,description,price,images,
> > categoriesStr,enddate_solar,begin_date_solar,status_solar,
> > current_stock_solar,retprice_solar,distprice_solar,
> > listprice_solar,mfgprice_solar,out_of_stock_solar,hide_
> > product_solar,saleprice_solar,metakey_solar,sales_enabled,
> > new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
> > FROM products"
> > deltaImportQuery="SELECT
> > orgidStr,idStr,name,code,description,price,images,
> > categoriesStr,enddate_solar,begin_date_solar,status_solar,
> > current_stock_solar,retprice_solar,distprice_solar,
> > listprice_solar,mfgprice_solar,out_of_stock_solar,hide_
> > product_solar,saleprice_solar,metakey_solar,sales_enabled,
> > new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
> > FROM products WHERE orgidStr = '${dataimporter.request.orgid}' AND
> idStr =
> > '${dataimporter.delta.idStr}'"
> > deltaQuery="SELECT idStr FROM products WHERE idStr =
> > '${dataimporter.request.prodidStr}' AND orgidStr =
> > '${dataimporter.request.orgid}'"
> > >
> > <field column="orgidStr" name="prod_orgidStr"/>
> > <field column="idStr" name="prod_idStr"/>
> > <field column="uuid" name="uuid"
> > template="org-${products.orgidStr}-prod-${products.idStr}"/>
> > <field column="name" name="prod_name"/>
> > <field column="code" name="prod_code"/>
> > <field column="description" name="description"/>
> > <field column="price" name="price"/>
> >
> >
> > ​This is the error I get :
> >
> > getNext() failed for query 'SELECT
> > orgidStr,idStr,name,code,description,price,images,
> > categoriesStr,enddate_solar,begin_date_solar,status_solar,
> > current_stock_solar,retprice_solar,distprice_solar,
> > listprice_solar,mfgprice_solar,out_of_stock_solar,hide_
> > product_solar,saleprice_solar,metakey_solar,sales_enabled,
> > new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
> > FROM products'
> >
> > :com.mongodb.MongoException$Network: can't call something : /
> > ​<<IP ADDRESS>>
> > :27017/
> > ​<<DB NAME>>
> >
> >
> >
> > Caused by: java.io.IOException: couldn't connect to [/
> > ​
> > ​<<IP ADDRESS>>:27017] bc:java.net.SocketTimeoutException: connect timed
> > out
> >
> > ​Have anyone else gone through this kind of issue ?
> >
> >
> >
> >
> > On Tue, Mar 28, 2017 at 6:20 PM, Rick Leir <rl...@leirtech.com> wrote:
> >
> > > Abhijit
> > > In Mongo you probably have one JSON record per document. You can post
> > that
> > > JSON record to Solr, and the JSON fields get indexed. The github
> project
> > > you mention does just that. If you use the Solr managed schema then
> Solr
> > > will automatically define fields based on what it receives. Otherwise
> you
> > > will need to carefully design a schema.xml.
> > > Cheers -- Rick
> > >
> > > On March 28, 2017 6:08:40 PM EDT, Abhijit Pawar <
> > > abhijit.ibizsoft@gmail.com> wrote:
> > > >Hello All,
> > > >
> > > >I am working on a requirement to index field of type JSON (in mongoDB
> > > >collection) in SOLR 5.4.0.
> > > >
> > > >I am using mongo-jdbc-dih which I found on GitHub :
> > > >
> > > >https://github.com/hrishik/solr-mongodb-dih
> > > >
> > > >However I could not find a fieldtype on Apache SOLR wiki page which
> > > >would
> > > >support JSON datatype in mongoDB.
> > > >
> > > >Can someone please recommend a way to include datatype / fieldtype in
> > > >SOLR
> > > >schema to support or index JSON data field from mongoDB.
> > > >Thanks.
> > > >
> > > >R​egards,
> > > >
> > > >Abhijit​
> > >
> > > --
> > > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> >
>

Re: Fieldtype json supported in SOLR 5.4.0 or 5.4.1

Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Did you upgrade your solrconfig.xml to the Solr 6.0 version too?
There are some difference in Solr 6.0 version which requires setting to
determine whether to use managed-schema or classic schema (the physical
schema.xml file)

Regards,
Edwin

On 1 April 2017 at 01:27, Abhijit Pawar <ab...@gmail.com> wrote:

> Hi Rick,
>
> I tried installing SOLR 6.0 since SOLR 6.0 has managed-schema and tried
> index the data from mongoDB :
>
>
> <dataConfig>
> <dataSource name="mongod" type="JdbcDataSource"
> driver="com.mongodb.jdbc.MongoDriver" url="mongodb://
> ​<<>IP-Address>​
> :27017/
> ​<<DB-NAME>>
> "/>
> <document name="products">
> <entity name="products"
> dataSource="mongod"
> transformer="TemplateTransformer,ProdsCatsFieldTransformer"
> onError="continue"
> pk="uuid"
> query="SELECT
> orgidStr,idStr,name,code,description,price,images,
> categoriesStr,enddate_solar,begin_date_solar,status_solar,
> current_stock_solar,retprice_solar,distprice_solar,
> listprice_solar,mfgprice_solar,out_of_stock_solar,hide_
> product_solar,saleprice_solar,metakey_solar,sales_enabled,
> new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
> FROM products"
> deltaImportQuery="SELECT
> orgidStr,idStr,name,code,description,price,images,
> categoriesStr,enddate_solar,begin_date_solar,status_solar,
> current_stock_solar,retprice_solar,distprice_solar,
> listprice_solar,mfgprice_solar,out_of_stock_solar,hide_
> product_solar,saleprice_solar,metakey_solar,sales_enabled,
> new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
> FROM products WHERE orgidStr = '${dataimporter.request.orgid}' AND idStr =
> '${dataimporter.delta.idStr}'"
> deltaQuery="SELECT idStr FROM products WHERE idStr =
> '${dataimporter.request.prodidStr}' AND orgidStr =
> '${dataimporter.request.orgid}'"
> >
> <field column="orgidStr" name="prod_orgidStr"/>
> <field column="idStr" name="prod_idStr"/>
> <field column="uuid" name="uuid"
> template="org-${products.orgidStr}-prod-${products.idStr}"/>
> <field column="name" name="prod_name"/>
> <field column="code" name="prod_code"/>
> <field column="description" name="description"/>
> <field column="price" name="price"/>
>
>
> ​This is the error I get :
>
> getNext() failed for query 'SELECT
> orgidStr,idStr,name,code,description,price,images,
> categoriesStr,enddate_solar,begin_date_solar,status_solar,
> current_stock_solar,retprice_solar,distprice_solar,
> listprice_solar,mfgprice_solar,out_of_stock_solar,hide_
> product_solar,saleprice_solar,metakey_solar,sales_enabled,
> new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
> FROM products'
>
> :com.mongodb.MongoException$Network: can't call something : /
> ​<<IP ADDRESS>>
> :27017/
> ​<<DB NAME>>
>
>
>
> Caused by: java.io.IOException: couldn't connect to [/
> ​
> ​<<IP ADDRESS>>:27017] bc:java.net.SocketTimeoutException: connect timed
> out
>
> ​Have anyone else gone through this kind of issue ?
>
>
>
>
> On Tue, Mar 28, 2017 at 6:20 PM, Rick Leir <rl...@leirtech.com> wrote:
>
> > Abhijit
> > In Mongo you probably have one JSON record per document. You can post
> that
> > JSON record to Solr, and the JSON fields get indexed. The github project
> > you mention does just that. If you use the Solr managed schema then Solr
> > will automatically define fields based on what it receives. Otherwise you
> > will need to carefully design a schema.xml.
> > Cheers -- Rick
> >
> > On March 28, 2017 6:08:40 PM EDT, Abhijit Pawar <
> > abhijit.ibizsoft@gmail.com> wrote:
> > >Hello All,
> > >
> > >I am working on a requirement to index field of type JSON (in mongoDB
> > >collection) in SOLR 5.4.0.
> > >
> > >I am using mongo-jdbc-dih which I found on GitHub :
> > >
> > >https://github.com/hrishik/solr-mongodb-dih
> > >
> > >However I could not find a fieldtype on Apache SOLR wiki page which
> > >would
> > >support JSON datatype in mongoDB.
> > >
> > >Can someone please recommend a way to include datatype / fieldtype in
> > >SOLR
> > >schema to support or index JSON data field from mongoDB.
> > >Thanks.
> > >
> > >R​egards,
> > >
> > >Abhijit​
> >
> > --
> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

Re: Fieldtype json supported in SOLR 5.4.0 or 5.4.1

Posted by Abhijit Pawar <ab...@gmail.com>.
Hi Rick,

I tried installing SOLR 6.0 since SOLR 6.0 has managed-schema and tried
index the data from mongoDB :


<dataConfig>
<dataSource name="mongod" type="JdbcDataSource"
driver="com.mongodb.jdbc.MongoDriver" url="mongodb://
​<<>IP-Address>​
:27017/
​<<DB-NAME>>
"/>
<document name="products">
<entity name="products"
dataSource="mongod"
transformer="TemplateTransformer,ProdsCatsFieldTransformer"
onError="continue"
pk="uuid"
query="SELECT
orgidStr,idStr,name,code,description,price,images,categoriesStr,enddate_solar,begin_date_solar,status_solar,current_stock_solar,retprice_solar,distprice_solar,listprice_solar,mfgprice_solar,out_of_stock_solar,hide_product_solar,saleprice_solar,metakey_solar,sales_enabled,new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
FROM products"
deltaImportQuery="SELECT
orgidStr,idStr,name,code,description,price,images,categoriesStr,enddate_solar,begin_date_solar,status_solar,current_stock_solar,retprice_solar,distprice_solar,listprice_solar,mfgprice_solar,out_of_stock_solar,hide_product_solar,saleprice_solar,metakey_solar,sales_enabled,new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
FROM products WHERE orgidStr = '${dataimporter.request.orgid}' AND idStr =
'${dataimporter.delta.idStr}'"
deltaQuery="SELECT idStr FROM products WHERE idStr =
'${dataimporter.request.prodidStr}' AND orgidStr =
'${dataimporter.request.orgid}'"
>
<field column="orgidStr" name="prod_orgidStr"/>
<field column="idStr" name="prod_idStr"/>
<field column="uuid" name="uuid"
template="org-${products.orgidStr}-prod-${products.idStr}"/>
<field column="name" name="prod_name"/>
<field column="code" name="prod_code"/>
<field column="description" name="description"/>
<field column="price" name="price"/>


​This is the error I get :

getNext() failed for query 'SELECT
orgidStr,idStr,name,code,description,price,images,categoriesStr,enddate_solar,begin_date_solar,status_solar,current_stock_solar,retprice_solar,distprice_solar,listprice_solar,mfgprice_solar,out_of_stock_solar,hide_product_solar,saleprice_solar,metakey_solar,sales_enabled,new_product,has_sku,configurable,rating,updatedAt,comparable,hide_price
FROM products'

:com.mongodb.MongoException$Network: can't call something : /
​<<IP ADDRESS>>
:27017/
​<<DB NAME>>


	
Caused by: java.io.IOException: couldn't connect to [/
​
​<<IP ADDRESS>>:27017] bc:java.net.SocketTimeoutException: connect timed out

​Have anyone else gone through this kind of issue ?




On Tue, Mar 28, 2017 at 6:20 PM, Rick Leir <rl...@leirtech.com> wrote:

> Abhijit
> In Mongo you probably have one JSON record per document. You can post that
> JSON record to Solr, and the JSON fields get indexed. The github project
> you mention does just that. If you use the Solr managed schema then Solr
> will automatically define fields based on what it receives. Otherwise you
> will need to carefully design a schema.xml.
> Cheers -- Rick
>
> On March 28, 2017 6:08:40 PM EDT, Abhijit Pawar <
> abhijit.ibizsoft@gmail.com> wrote:
> >Hello All,
> >
> >I am working on a requirement to index field of type JSON (in mongoDB
> >collection) in SOLR 5.4.0.
> >
> >I am using mongo-jdbc-dih which I found on GitHub :
> >
> >https://github.com/hrishik/solr-mongodb-dih
> >
> >However I could not find a fieldtype on Apache SOLR wiki page which
> >would
> >support JSON datatype in mongoDB.
> >
> >Can someone please recommend a way to include datatype / fieldtype in
> >SOLR
> >schema to support or index JSON data field from mongoDB.
> >Thanks.
> >
> >R​egards,
> >
> >Abhijit​
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Fieldtype json supported in SOLR 5.4.0 or 5.4.1

Posted by Abhijit Pawar <ap...@ibizsoftinc.com>.
Thanks Rick.

Does that mean I need to define managed-schema.xml, I thought it gets
created by default on installing but only on later versions of  SOLR ( 6.0
or later).

Will managed-schema help in indexing the JSON type fields in the mongoDB ?
How do I define the managed-schema in SOLR 5.4.0 ?



Best Regards,
<http://www.ibizsoftinc.com/>
<http://www.ibizsoftinc.com/>
Abhijit Pawar
Office : +1 (469) 287 2005 x 110



Follow us on:





On Tue, Mar 28, 2017 at 6:20 PM, Rick Leir <rl...@leirtech.com> wrote:

> Abhijit
> In Mongo you probably have one JSON record per document. You can post that
> JSON record to Solr, and the JSON fields get indexed. The github project
> you mention does just that. If you use the Solr managed schema then Solr
> will automatically define fields based on what it receives. Otherwise you
> will need to carefully design a schema.xml.
> Cheers -- Rick
>
> On March 28, 2017 6:08:40 PM EDT, Abhijit Pawar <
> abhijit.ibizsoft@gmail.com> wrote:
> >Hello All,
> >
> >I am working on a requirement to index field of type JSON (in mongoDB
> >collection) in SOLR 5.4.0.
> >
> >I am using mongo-jdbc-dih which I found on GitHub :
> >
> >https://github.com/hrishik/solr-mongodb-dih
> >
> >However I could not find a fieldtype on Apache SOLR wiki page which
> >would
> >support JSON datatype in mongoDB.
> >
> >Can someone please recommend a way to include datatype / fieldtype in
> >SOLR
> >schema to support or index JSON data field from mongoDB.
> >Thanks.
> >
> >R​egards,
> >
> >Abhijit​
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Fieldtype json supported in SOLR 5.4.0 or 5.4.1

Posted by Rick Leir <rl...@leirtech.com>.
Abhijit 
In Mongo you probably have one JSON record per document. You can post that JSON record to Solr, and the JSON fields get indexed. The github project you mention does just that. If you use the Solr managed schema then Solr will automatically define fields based on what it receives. Otherwise you will need to carefully design a schema.xml.
Cheers -- Rick

On March 28, 2017 6:08:40 PM EDT, Abhijit Pawar <ab...@gmail.com> wrote:
>Hello All,
>
>I am working on a requirement to index field of type JSON (in mongoDB
>collection) in SOLR 5.4.0.
>
>I am using mongo-jdbc-dih which I found on GitHub :
>
>https://github.com/hrishik/solr-mongodb-dih
>
>However I could not find a fieldtype on Apache SOLR wiki page which
>would
>support JSON datatype in mongoDB.
>
>Can someone please recommend a way to include datatype / fieldtype in
>SOLR
>schema to support or index JSON data field from mongoDB.
>Thanks.
>
>R​egards,
>
>Abhijit​

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.