You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Naska Osmani <na...@gmail.com> on 2013/05/25 08:49:35 UTC
Indexing Solr, Multiple Doc Types. Production of Multiple Values
for UniqueKey Field Using TemplateTransformer
Hello,
I want to index multiple tables of a database into a single solr index.
I used TemplateTransformer to concatenate a prefix, the id of the table
or entity with the uniqueKey uid so that entities don't overwrite
eachother. The documents don't get indexed with the error message:
org.apache.solr.common.SolrException: Document contains multiple values
for uniqueKey field: uid=[A_1, dc1999fcf12df900]
at
org.apache.solr.update.AddUpdateCommand.getIndexedId(AddUpdateCommand.java:91)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:464)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at
org.apache.solr.update.processor.SignatureUpdateProcessorFactory$SignatureUpdateProcessor.processAdd(SignatureUpdateProcessorFactory.java:194)
at
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70)
at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:235)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:500)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:319)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:227)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)
Here is part of my schema.xml:
<fields>
<field name="aid" type="string" multiValued="true"/>
<field name="aname" type="string" indexed="true" stored="true"
omitNorms="true" termVectors="true" multiValued="true"/>
<field name="acountry" type="string" indexed="true" stored="true"
omitNorms="true" termVectors="true" multivalued="true" />
<field name="bid" type="string" multiValued="true"/>
<field name="bname" type="string" indexed="true" stored="true"
omitNorms="true" termVectors="true" multiValued="true"/>
<field name="bcountry" type="string" indexed="true" stored="true"
omitNorms="true" termVectors="true" multivalued="true" />
<field name="uid" type="string"/>
<field name="doc_type" type="string"/>
<field name="allText" type="text_general" indexed="true"
stored="true" multiValued="true" omitNorms="true" termVectors="true" />
</fields>
<uniqueKey>uid</uniqueKey>
And my data-config.xml:
<document name="doc">
<entity name="atest" pk="id" transformer="TemplateTransformer"
query="SELECT id, name, country FROM atests">
<field column="uid" name="uid" template="A_${atest.id}"/>
<field column="doc_type" template="ATEST"/>
<field column="id" name="aid"/>
<field column="name" name="aname"/>
<field column="country" name="acountry"/>
</entity>
<entity name="btest" pk="id" transformer="TemplateTransformer"
query="SELECT id, name, country FROM btests">
<field column="uid" template="B_${btest.id}"/>
<field column="doc_type" template="BTEST"/>
<field column="id" name="bid"/>
<field column="name" name="bname"/>
<field column="country" name="bcountry"/>
</entity>
</document>
I have tried to set multivalued to true or false or earase it in aid and
bid fields but this didn't solve the issue.
Thanks in advance
Indexing Solr, Multiple Doc Types. Production of Multiple Values
for UniqueKey Field Using TemplateTransformer
Posted by Naska Osmani <na...@gmail.com>.
Hello,
I want to index multiple tables of a database into a single solr index.
I used TemplateTransformer to concatenate a prefix, the id of the table
or entity with the uniqueKey uid so that entities don't overwrite
eachother. The documents don't get indexed with the error message:
org.apache.solr.common.SolrException: Document contains multiple values
for uniqueKey field: uid=[A_1, dc1999fcf12df900]
at
org.apache.solr.update.AddUpdateCommand.getIndexedId(AddUpdateCommand.java:91)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:464)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at
org.apache.solr.update.processor.SignatureUpdateProcessorFactory$SignatureUpdateProcessor.processAdd(SignatureUpdateProcessorFactory.java:194)
at
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70)
at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:235)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:500)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:319)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:227)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)
Here is part of my schema.xml:
<fields>
<field name="aid" type="string" multiValued="true"/>
<field name="aname" type="string" indexed="true" stored="true"
omitNorms="true" termVectors="true" multiValued="true"/>
<field name="acountry" type="string" indexed="true" stored="true"
omitNorms="true" termVectors="true" multivalued="true" />
<field name="bid" type="string" multiValued="true"/>
<field name="bname" type="string" indexed="true" stored="true"
omitNorms="true" termVectors="true" multiValued="true"/>
<field name="bcountry" type="string" indexed="true" stored="true"
omitNorms="true" termVectors="true" multivalued="true" />
<field name="uid" type="string"/>
<field name="doc_type" type="string"/>
<field name="allText" type="text_general" indexed="true"
stored="true" multiValued="true" omitNorms="true" termVectors="true" />
</fields>
<uniqueKey>uid</uniqueKey>
And my data-config.xml:
<document name="doc">
<entity name="atest" pk="id" transformer="TemplateTransformer"
query="SELECT id, name, country FROM atests">
<field column="uid" name="uid" template="A_${atest.id}"/>
<field column="doc_type" template="ATEST"/>
<field column="id" name="aid"/>
<field column="name" name="aname"/>
<field column="country" name="acountry"/>
</entity>
<entity name="btest" pk="id" transformer="TemplateTransformer"
query="SELECT id, name, country FROM btests">
<field column="uid" template="B_${btest.id}"/>
<field column="doc_type" template="BTEST"/>
<field column="id" name="bid"/>
<field column="name" name="bname"/>
<field column="country" name="bcountry"/>
</entity>
</document>
I have tried to set multivalued to true or false or earase it in aid and
bid fields but this didn't solve the issue.
Thanks in advance
Re: Indexing Solr, Multiple Doc Types. Production of Multiple Values
for UniqueKey Field Using TemplateTransformer
Posted by Chris Hostetter <ho...@fucit.org>.
: org.apache.solr.common.SolrException: Document contains multiple values for
: uniqueKey field: uid=[A_1, dc1999fcf12df900]
By the looks of things, your TemplateTransformer is properly creating a
value of "A_${atest.id}" where "${atest.id} == 1" for that document ...
the problem seems to be that somehow another value is getting put in your
uid field containing "dc1999fcf12df900"
Based on your stack trace, i suspect that in addition to having DIH
create a value for your "uid" field, you also have
SignatureUpdateProcessorFactory configured (in your solrconfig.xml) to
generate a synthetic unique id based on the signature of some fields as
well...
: org.apache.solr.update.processor.SignatureUpdateProcessorFactory$SignatureUpdateProcessor.processAdd(SignatureUpdateProcessorFactory.java:194)
-Hoss