You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Verthosa <ge...@gmail.com> on 2018/11/09 12:50:33 UTC

Sql server data import

Hello, i managed to set up a connection to my sql server to import data into
Solr. The idea is to import filetables but for now i first want to get it
working using regular tables. So i created 

*data-config.xml*
<dataConfig>   
  <dataSource type="JdbcDataSource"    
     driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"    
    
url="jdbc:sqlserver://localhost;databaseName=inConnexion_Tenant2;integratedSecurity=true"
/>   
  <document>   
   <entity name="Dashboard" pk="id" query="SELECT Id,PublicId FROM
foundation.Shops">   
	 <field column="Id" name="Id"/>   
     <field column="PublicId" name="PublicId" /> 
   </entity>   
  </document>   
  </dataConfig>  

*schema.xml*
i added
<field name="Id" type="string" indexed="true" stored="true" required="true"
multiValued="false" />	
<field name="PublicId" type="string" indexed="true" stored="true"
multiValued="false"/>

and changed uniqueKey entry to 
<uniqueKey>Id</uniqueKey>

When i want to import my data (which is just data like Id: 5, PublicId:
"test"), i get the following error in the logging. 

Error creating document : SolrInputDocument(fields: [PublicId=10065,&#8203;
Id=117])


I tried all sorts of things but can't get it fixed. Is anyone want to give
me a hand?

thanks in advance!




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Sql server data import

Posted by Erick Erickson <er...@gmail.com>.

Ok, what that means is you're letting Solr do its best to figure out
what fields you should have in the schema and how they're defined.
Almost invariably, you can do better by explicitly defining the fields
you need in your schema rather than enabling add-unknown----. It's
fine for getting started, but not advised for production.

Best,
Erick
On Fri, Nov 9, 2018 at 7:52 AM Verthosa <ge...@gmail.com> wrote:
>
> Hello, i managed to fix the problem. I'm using Solr 7.5.0. My problem was
> that in the server logs i got "This Indexschema is not mutable" (i did not
> know about the logs folder, so i just found out 5 minutes ago). I fixed it
> by modifying solrconfig.xml to
>
> <updateRequestProcessorChain
> name=&quot;add-unknown-fields-to-the-schema&quot;
> default=&quot;${update.autoCreateFields:&lt;b>false*}"
>
> processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
>     <processor class="solr.LogUpdateProcessorFactory"/>
>     <processor class="solr.DistributedUpdateProcessorFactory"/>
>     <processor class="solr.RunUpdateProcessorFactory"/>
> </updateRequestProcessorChain>
>
> Since then the indexing is done correctly. I even got the blob fields
> indexation working now ! Thanks for your reply, everything is fixed for now.
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Sql server data import

Posted by Verthosa <ge...@gmail.com>.

Hello, i managed to fix the problem. I'm using Solr 7.5.0. My problem was
that in the server logs i got "This Indexschema is not mutable" (i did not
know about the logs folder, so i just found out 5 minutes ago). I fixed it
by modifying solrconfig.xml to

<updateRequestProcessorChain
name=&quot;add-unknown-fields-to-the-schema&quot;
default=&quot;${update.autoCreateFields:&lt;b>false*}"
                            
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
    <processor class="solr.LogUpdateProcessorFactory"/>
    <processor class="solr.DistributedUpdateProcessorFactory"/>
    <processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>

Since then the indexing is done correctly. I even got the blob fields
indexation working now ! Thanks for your reply, everything is fixed for now. 




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Sql server data import

Posted by Alexandre Rafalovitch <ar...@gmail.com>.

Which version of Solr is it? Because we have not used schema.xml for a
very long time. It has been managed-schema instead.

Also, have you tried using DIH example that uses database and
modifying it just enough to read data from your database. Even if it
has a lot of extra junk, this would test half of the pipeline, which
you can then transfer to the clean setup.

Regards,
   Alex.
On Fri, 9 Nov 2018 at 08:09, Verthosa <ge...@gmail.com> wrote:
>
> Hello, i managed to set up a connection to my sql server to import data into
> Solr. The idea is to import filetables but for now i first want to get it
> working using regular tables. So i created
>
> *data-config.xml*
> <dataConfig>
>   <dataSource type="JdbcDataSource"
>      driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
>
> url="jdbc:sqlserver://localhost;databaseName=inConnexion_Tenant2;integratedSecurity=true"
> />
>   <document>
>    <entity name="Dashboard" pk="id" query="SELECT Id,PublicId FROM
> foundation.Shops">
>          <field column="Id" name="Id"/>
>      <field column="PublicId" name="PublicId" />
>    </entity>
>   </document>
>   </dataConfig>
>
> *schema.xml*
> i added
> <field name="Id" type="string" indexed="true" stored="true" required="true"
> multiValued="false" />
> <field name="PublicId" type="string" indexed="true" stored="true"
> multiValued="false"/>
>
> and changed uniqueKey entry to
> <uniqueKey>Id</uniqueKey>
>
> When i want to import my data (which is just data like Id: 5, PublicId:
> "test"), i get the following error in the logging.
>
> Error creating document : SolrInputDocument(fields: [PublicId=10065,&#8203;
> Id=117])
>
>
> I tried all sorts of things but can't get it fixed. Is anyone want to give
> me a hand?
>
> thanks in advance!
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: Sql server data import

Posted by "Gu, Steve (CDC/DDPHSS/OS) (CTR)" <cz...@cdc.gov.INVALID>.

What is "&#8203;"  in the PublicId?  Is it part of the data?  Did you check if the special characters in your data cause the problem?

Steve

###########
Error creating document : SolrInputDocument(fields: [PublicId=10065,&#8203;
Id=117])

-----Original Message-----
From: Verthosa <ge...@gmail.com> 
Sent: Friday, November 9, 2018 7:51 AM
To: solr-user@lucene.apache.org
Subject: Sql server data import

Hello, i managed to set up a connection to my sql server to import data into Solr. The idea is to import filetables but for now i first want to get it working using regular tables. So i created 

*data-config.xml*
<dataConfig>   
  <dataSource type="JdbcDataSource"    
     driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"    
    
url="jdbc:sqlserver://localhost;databaseName=inConnexion_Tenant2;integratedSecurity=true"
/>   
  <document>   
   <entity name="Dashboard" pk="id" query="SELECT Id,PublicId FROM
foundation.Shops">   
	 <field column="Id" name="Id"/>   
     <field column="PublicId" name="PublicId" /> 
   </entity>   
  </document>   
  </dataConfig>  

*schema.xml*
i added
<field name="Id" type="string" indexed="true" stored="true" required="true"
multiValued="false" />	
<field name="PublicId" type="string" indexed="true" stored="true"
multiValued="false"/>

and changed uniqueKey entry to
<uniqueKey>Id</uniqueKey>

When i want to import my data (which is just data like Id: 5, PublicId:
"test"), i get the following error in the logging. 

Error creating document : SolrInputDocument(fields: [PublicId=10065,&#8203;
Id=117])


I tried all sorts of things but can't get it fixed. Is anyone want to give me a hand?

thanks in advance!




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html