You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by KARTHIK SHIVAKUMAR <ns...@gmail.com> on 2021/10/01 16:51:37 UTC
Solr User with "Document is missing mandatory uniqueKey"
Hi
Spec : Jdk 15, Solr 8.8.1
Problem : Error from server at http://localhost:8081/solr/Log-Test: Document is missing mandatory uniqueKey field: id.
Process : I have written a small Solr-Client code to push content to solr-indexer on port 8081
Question : I have encountered this problem for the first time and did not find any proper solutions on www for the same.
Request : How to fix this With-Out introducing 'id' as another parameter passed on from content side ??.
Thx for the same.
Re: Solr User with "Document is missing mandatory uniqueKey"
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I _think_ uniqueKey declaration is optional. So, both id field and
uniqueKey declaration could be removed. But then, something in
solrconfig.xml will also need to be adjusted as other parts of Solr
functionality will need to be disabled as well, that do rely on unique
key semantics.
UpdateRequestProcessor is a bit harder to setup, but is probably a
better fit longer term.
Of course, if you are doing Search index creation from Java and you
don't need cloud/advanced functionality, maybe you don't need most of
Solr and using Lucene directly will fit your needs better.
Regards,
Alex.
On Wed, 6 Oct 2021 at 11:21, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 10/6/21 3:39 AM, KARTHIK SHIVAKUMAR wrote:
>
> > Why Solr is not considering this field a internal to it's objective and allowing to be declared...
>
> I do not understand what you are asking here.
>
> > Solr should handle this Id with out exposing , so the same is not reflected by users to raise this case again and again....
> >
> > If an user can directly index / search with out this field (forced to add )... since if solr can handle this internally for each doc created/ deleted.
>
> Some users might be perfectly OK with Solr always handling the uniqueKey
> itself. But I can guarantee you that an enormous fraction of the user
> base would not want Solr to do that.
>
> We do provide a way for users to do what you want -- have Solr create
> the ID value when one is not provided:
>
> https://solr.apache.org/docs/8_10_0/solr-core/org/apache/solr/update/processor/UUIDUpdateProcessorFactory.html
>
> If you need help incorporating that update processor into your config,
> then we would need to see your solrconfig.xml as well as your schema,
> and then we may need to ask you a few questions to tailor the response
> for your situation. Exactly what filename will contain your schema can
> vary, most often it will be "managed-schema" with no extension. If you
> are running in cloud mode, then the active config and schema will be
> stored in zookeeper, not on the disk.
>
> Thanks,
> Shawn
>
Re: Solr User with "Document is missing mandatory uniqueKey"
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/6/21 3:39 AM, KARTHIK SHIVAKUMAR wrote:
> Why Solr is not considering this field a internal to it's objective and allowing to be declared...
I do not understand what you are asking here.
> Solr should handle this Id with out exposing , so the same is not reflected by users to raise this case again and again....
>
> If an user can directly index / search with out this field (forced to add )... since if solr can handle this internally for each doc created/ deleted.
Some users might be perfectly OK with Solr always handling the uniqueKey
itself. But I can guarantee you that an enormous fraction of the user
base would not want Solr to do that.
We do provide a way for users to do what you want -- have Solr create
the ID value when one is not provided:
https://solr.apache.org/docs/8_10_0/solr-core/org/apache/solr/update/processor/UUIDUpdateProcessorFactory.html
If you need help incorporating that update processor into your config,
then we would need to see your solrconfig.xml as well as your schema,
and then we may need to ask you a few questions to tailor the response
for your situation. Exactly what filename will contain your schema can
vary, most often it will be "managed-schema" with no extension. If you
are running in cloud mode, then the active config and schema will be
stored in zookeeper, not on the disk.
Thanks,
Shawn
Re: Solr User with "Document is missing mandatory uniqueKey"
Posted by Dave <ha...@gmail.com>.
Also a unique id is valuable if for example you are indexing from a database, and you use the id from the table, but of course other tables you index can have the same id value, so your indexer can append it with the table name as a simple example. I can’t think of any situation where you would not want this key,
> On Oct 6, 2021, at 8:54 AM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>
> Because it 99% of the use cases want to update solr index with changes.
> Which requires some way of duplicate management.
>
> But Solr does support a one-shot 'don't care' method as well, just as the
> previous reply showed.
>
> Try that and let us know if you are still stuck.
>
> Regards,
> Alex
>
>> On Wed., Oct. 6, 2021, 6:37 a.m. KARTHIK SHIVAKUMAR, <ns...@gmail.com>
>> wrote:
>>
>> Hi
>>
>> Why Solr is not considering this field a internal to it's objective and
>> allowing to be declared...
>>
>> Solr should handle this Id with out exposing , so the same is not
>> reflected by users to raise this case again and again....
>>
>> If an user can directly index / search with out this field (forced to add
>> )... since if solr can handle this internally for each doc created/ deleted.
>>
>> with regards
>> karthik
>>
>>> On 2021/10/01 20:09:59, Shawn Heisey <ap...@elyograg.org> wrote:
>>> On 10/1/21 10:51 AM, KARTHIK SHIVAKUMAR wrote:
>>>> Problem : Error from server at http://localhost:8081/solr/Log-Test:
>> Document is missing mandatory uniqueKey field: id.
>>>>
>>>> Process : I have written a small Solr-Client code to push content to
>> solr-indexer on port 8081
>>>>
>>>> Question : I have encountered this problem for the first time and did
>> not find any proper solutions on www for the same.
>>>>
>>>> Request : How to fix this With-Out introducing 'id' as another
>> parameter passed on from content side ??.
>>>
>>> The entire point of uniqueKey is that every document will have something
>>> in that field, and that if an existing document has the same value in
>>> that field as a document that is being indexed, it is deleted and the
>>> new document takes its place. By design, if you have a uniqueKey field,
>>> every submitted document must have a value in that field. A uniqueKey
>>> is required for several Solr features, including distributed indexes
>>> (those with more than one shard).
>>>
>>> There is an existing update processor that can fill in a field with a
>>> random UUID if the field doesn't exist in the submitted document. But
>>> note that if you do use this, you will no longer be able to have Solr
>>> automatically replace that old document with a new one without some
>>> extra work, because nothing outside of Solr will have any knowledge
>>> about what the uniqueKey field contains for that document.
>>>
>>> I hope that gives you a direction.
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>
Re: Solr User with "Document is missing mandatory uniqueKey"
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Because it 99% of the use cases want to update solr index with changes.
Which requires some way of duplicate management.
But Solr does support a one-shot 'don't care' method as well, just as the
previous reply showed.
Try that and let us know if you are still stuck.
Regards,
Alex
On Wed., Oct. 6, 2021, 6:37 a.m. KARTHIK SHIVAKUMAR, <ns...@gmail.com>
wrote:
> Hi
>
> Why Solr is not considering this field a internal to it's objective and
> allowing to be declared...
>
> Solr should handle this Id with out exposing , so the same is not
> reflected by users to raise this case again and again....
>
> If an user can directly index / search with out this field (forced to add
> )... since if solr can handle this internally for each doc created/ deleted.
>
> with regards
> karthik
>
> On 2021/10/01 20:09:59, Shawn Heisey <ap...@elyograg.org> wrote:
> > On 10/1/21 10:51 AM, KARTHIK SHIVAKUMAR wrote:
> > > Problem : Error from server at http://localhost:8081/solr/Log-Test:
> Document is missing mandatory uniqueKey field: id.
> > >
> > > Process : I have written a small Solr-Client code to push content to
> solr-indexer on port 8081
> > >
> > > Question : I have encountered this problem for the first time and did
> not find any proper solutions on www for the same.
> > >
> > > Request : How to fix this With-Out introducing 'id' as another
> parameter passed on from content side ??.
> >
> > The entire point of uniqueKey is that every document will have something
> > in that field, and that if an existing document has the same value in
> > that field as a document that is being indexed, it is deleted and the
> > new document takes its place. By design, if you have a uniqueKey field,
> > every submitted document must have a value in that field. A uniqueKey
> > is required for several Solr features, including distributed indexes
> > (those with more than one shard).
> >
> > There is an existing update processor that can fill in a field with a
> > random UUID if the field doesn't exist in the submitted document. But
> > note that if you do use this, you will no longer be able to have Solr
> > automatically replace that old document with a new one without some
> > extra work, because nothing outside of Solr will have any knowledge
> > about what the uniqueKey field contains for that document.
> >
> > I hope that gives you a direction.
> >
> > Thanks,
> > Shawn
> >
> >
>
Re: Solr User with "Document is missing mandatory uniqueKey"
Posted by KARTHIK SHIVAKUMAR <ns...@gmail.com>.
Hi
Why Solr is not considering this field a internal to it's objective and allowing to be declared...
Solr should handle this Id with out exposing , so the same is not reflected by users to raise this case again and again....
If an user can directly index / search with out this field (forced to add )... since if solr can handle this internally for each doc created/ deleted.
with regards
karthik
On 2021/10/01 20:09:59, Shawn Heisey <ap...@elyograg.org> wrote:
> On 10/1/21 10:51 AM, KARTHIK SHIVAKUMAR wrote:
> > Problem : Error from server at http://localhost:8081/solr/Log-Test: Document is missing mandatory uniqueKey field: id.
> >
> > Process : I have written a small Solr-Client code to push content to solr-indexer on port 8081
> >
> > Question : I have encountered this problem for the first time and did not find any proper solutions on www for the same.
> >
> > Request : How to fix this With-Out introducing 'id' as another parameter passed on from content side ??.
>
> The entire point of uniqueKey is that every document will have something
> in that field, and that if an existing document has the same value in
> that field as a document that is being indexed, it is deleted and the
> new document takes its place. By design, if you have a uniqueKey field,
> every submitted document must have a value in that field. A uniqueKey
> is required for several Solr features, including distributed indexes
> (those with more than one shard).
>
> There is an existing update processor that can fill in a field with a
> random UUID if the field doesn't exist in the submitted document. But
> note that if you do use this, you will no longer be able to have Solr
> automatically replace that old document with a new one without some
> extra work, because nothing outside of Solr will have any knowledge
> about what the uniqueKey field contains for that document.
>
> I hope that gives you a direction.
>
> Thanks,
> Shawn
>
>
Re: Solr User with "Document is missing mandatory uniqueKey"
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/1/21 10:51 AM, KARTHIK SHIVAKUMAR wrote:
> Problem : Error from server at http://localhost:8081/solr/Log-Test: Document is missing mandatory uniqueKey field: id.
>
> Process : I have written a small Solr-Client code to push content to solr-indexer on port 8081
>
> Question : I have encountered this problem for the first time and did not find any proper solutions on www for the same.
>
> Request : How to fix this With-Out introducing 'id' as another parameter passed on from content side ??.
The entire point of uniqueKey is that every document will have something
in that field, and that if an existing document has the same value in
that field as a document that is being indexed, it is deleted and the
new document takes its place. By design, if you have a uniqueKey field,
every submitted document must have a value in that field. A uniqueKey
is required for several Solr features, including distributed indexes
(those with more than one shard).
There is an existing update processor that can fill in a field with a
random UUID if the field doesn't exist in the submitted document. But
note that if you do use this, you will no longer be able to have Solr
automatically replace that old document with a new one without some
extra work, because nothing outside of Solr will have any knowledge
about what the uniqueKey field contains for that document.
I hope that gives you a direction.
Thanks,
Shawn