You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alexandre Rafalovitch <ar...@gmail.com> on 2014/07/01 11:50:36 UTC
Restriction on type of uniqueKey field?
Hello,
I remember reading somewhere that id field (uniqueKey) must be String.
But I cannot find the definitive confirmation, just that it should be
non-analyzed.
Can I use a single-valued TrieLongField type, with precision set to 0?
Or am I going to hit issues?
Regards,
Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency
Re: Restriction on type of uniqueKey field?
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/1/2014 8:43 AM, Koji Sekiguchi wrote:
> In addition, KeywordTokenizer can be seemingly used but it should be
> avoided
> for unique key field. One of my customers that used it and they had got OOM
> during a long term indexing. As it was difficult to find the problem,
> I'd like to share my experience.
Aside from Koji's experience, there is another reason that a TextField
with something like KeywordTokenizer/LowerCaseFilter should not be used
for the uniqueKey. It is documented in the wiki:
https://wiki.apache.org/solr/UniqueKey#Text_field_in_the_document
Thanks,
Shawn
Re: Restriction on type of uniqueKey field?
Posted by Erick Erickson <er...@gmail.com>.
non-String <uniqueKey> fields have historically popped out
in weird places. I think at one point, for instance,
QueryElevationComponent barfed on non-string types.
So, there may still be edge cases in which this can be a problem.
IMO, they're all bugs though.
Erick
On Tue, Jul 1, 2014 at 7:43 AM, Koji Sekiguchi <ko...@r.email.ne.jp> wrote:
> In addition, KeywordTokenizer can be seemingly used but it should be avoided
> for unique key field. One of my customers that used it and they had got OOM
> during a long term indexing. As it was difficult to find the problem,
> I'd like to share my experience.
>
> Koji
> --
> http://soleami.com/blog/comparing-document-classification-functions-of-lucene-and-mahout.html
>
>
> (2014/07/01 6:48), Alexandre Rafalovitch wrote:
>>
>> I wasn't thinking of shard keys, but may have been confused in the
>> reading.
>>
>> Thank you everyone, the long key is working just fine for me.
>>
>> Regards,
>> Alex.
>> Personal website: http://www.outerthoughts.com/
>> Current project: http://www.solr-start.com/ - Accelerating your Solr
>> proficiency
>>
>>
>> On Tue, Jul 1, 2014 at 8:15 PM, Michael Della Bitta
>> <mi...@appinions.com> wrote:
>>>
>>> Alex, maybe you're thinking of constraints put on shard keys?
>>>
>>> Michael Della Bitta
>>>
>>> Applications Developer
>>>
>>> o: +1 646 532 3062
>>>
>>> appinions inc.
>>>
>>> “The Science of Influence Marketing”
>>>
>>> 18 East 41st Street
>>>
>>> New York, NY 10017
>>>
>>> t: @appinions <https://twitter.com/Appinions> | g+:
>>> plus.google.com/appinions
>>>
>>> <https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
>>> w: appinions.com <http://www.appinions.com/>
>>>
>>>
>>> On Tue, Jul 1, 2014 at 7:05 AM, Shalin Shekhar Mangar <
>>> shalinmangar@gmail.com> wrote:
>>>
>>>> No, you definitely can have an int or long uniqueKey. A lot of Solr's
>>>> tests
>>>> use such a uniqueKey. See
>>>> solr/core/src/test-files/solr/collection1/conf/schema.xml
>>>>
>>>>
>>>> On Tue, Jul 1, 2014 at 3:20 PM, Alexandre Rafalovitch
>>>> <ar...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I remember reading somewhere that id field (uniqueKey) must be String.
>>>>> But I cannot find the definitive confirmation, just that it should be
>>>>> non-analyzed.
>>>>>
>>>>> Can I use a single-valued TrieLongField type, with precision set to 0?
>>>>> Or am I going to hit issues?
>>>>>
>>>>> Regards,
>>>>> Alex.
>>>>> Personal website: http://www.outerthoughts.com/
>>>>> Current project: http://www.solr-start.com/ - Accelerating your Solr
>>>>> proficiency
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>
>
>
>
Re: Restriction on type of uniqueKey field?
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
In addition, KeywordTokenizer can be seemingly used but it should be avoided
for unique key field. One of my customers that used it and they had got OOM
during a long term indexing. As it was difficult to find the problem,
I'd like to share my experience.
Koji
--
http://soleami.com/blog/comparing-document-classification-functions-of-lucene-and-mahout.html
(2014/07/01 6:48), Alexandre Rafalovitch wrote:
> I wasn't thinking of shard keys, but may have been confused in the reading.
>
> Thank you everyone, the long key is working just fine for me.
>
> Regards,
> Alex.
> Personal website: http://www.outerthoughts.com/
> Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency
>
>
> On Tue, Jul 1, 2014 at 8:15 PM, Michael Della Bitta
> <mi...@appinions.com> wrote:
>> Alex, maybe you're thinking of constraints put on shard keys?
>>
>> Michael Della Bitta
>>
>> Applications Developer
>>
>> o: +1 646 532 3062
>>
>> appinions inc.
>>
>> “The Science of Influence Marketing”
>>
>> 18 East 41st Street
>>
>> New York, NY 10017
>>
>> t: @appinions <https://twitter.com/Appinions> | g+:
>> plus.google.com/appinions
>> <https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
>> w: appinions.com <http://www.appinions.com/>
>>
>>
>> On Tue, Jul 1, 2014 at 7:05 AM, Shalin Shekhar Mangar <
>> shalinmangar@gmail.com> wrote:
>>
>>> No, you definitely can have an int or long uniqueKey. A lot of Solr's tests
>>> use such a uniqueKey. See
>>> solr/core/src/test-files/solr/collection1/conf/schema.xml
>>>
>>>
>>> On Tue, Jul 1, 2014 at 3:20 PM, Alexandre Rafalovitch <ar...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I remember reading somewhere that id field (uniqueKey) must be String.
>>>> But I cannot find the definitive confirmation, just that it should be
>>>> non-analyzed.
>>>>
>>>> Can I use a single-valued TrieLongField type, with precision set to 0?
>>>> Or am I going to hit issues?
>>>>
>>>> Regards,
>>>> Alex.
>>>> Personal website: http://www.outerthoughts.com/
>>>> Current project: http://www.solr-start.com/ - Accelerating your Solr
>>>> proficiency
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>
Re: Restriction on type of uniqueKey field?
Posted by Jack Krupansky <ja...@basetechnology.com>.
My vague recollection is that at least at one time there was a limitation
somewhere in SolrCloud, but whether that is still true, I don't know.
-- Jack Krupansky
-----Original Message-----
From: Alexandre Rafalovitch
Sent: Tuesday, July 1, 2014 9:48 AM
To: solr-user@lucene.apache.org
Subject: Re: Restriction on type of uniqueKey field?
I wasn't thinking of shard keys, but may have been confused in the reading.
Thank you everyone, the long key is working just fine for me.
Regards,
Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr
proficiency
On Tue, Jul 1, 2014 at 8:15 PM, Michael Della Bitta
<mi...@appinions.com> wrote:
> Alex, maybe you're thinking of constraints put on shard keys?
>
> Michael Della Bitta
>
> Applications Developer
>
> o: +1 646 532 3062
>
> appinions inc.
>
> “The Science of Influence Marketing”
>
> 18 East 41st Street
>
> New York, NY 10017
>
> t: @appinions <https://twitter.com/Appinions> | g+:
> plus.google.com/appinions
> <https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
> w: appinions.com <http://www.appinions.com/>
>
>
> On Tue, Jul 1, 2014 at 7:05 AM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
>
>> No, you definitely can have an int or long uniqueKey. A lot of Solr's
>> tests
>> use such a uniqueKey. See
>> solr/core/src/test-files/solr/collection1/conf/schema.xml
>>
>>
>> On Tue, Jul 1, 2014 at 3:20 PM, Alexandre Rafalovitch
>> <ar...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I remember reading somewhere that id field (uniqueKey) must be String.
>> > But I cannot find the definitive confirmation, just that it should be
>> > non-analyzed.
>> >
>> > Can I use a single-valued TrieLongField type, with precision set to 0?
>> > Or am I going to hit issues?
>> >
>> > Regards,
>> > Alex.
>> > Personal website: http://www.outerthoughts.com/
>> > Current project: http://www.solr-start.com/ - Accelerating your Solr
>> > proficiency
>> >
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
Re: Restriction on type of uniqueKey field?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I wasn't thinking of shard keys, but may have been confused in the reading.
Thank you everyone, the long key is working just fine for me.
Regards,
Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency
On Tue, Jul 1, 2014 at 8:15 PM, Michael Della Bitta
<mi...@appinions.com> wrote:
> Alex, maybe you're thinking of constraints put on shard keys?
>
> Michael Della Bitta
>
> Applications Developer
>
> o: +1 646 532 3062
>
> appinions inc.
>
> “The Science of Influence Marketing”
>
> 18 East 41st Street
>
> New York, NY 10017
>
> t: @appinions <https://twitter.com/Appinions> | g+:
> plus.google.com/appinions
> <https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
> w: appinions.com <http://www.appinions.com/>
>
>
> On Tue, Jul 1, 2014 at 7:05 AM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
>
>> No, you definitely can have an int or long uniqueKey. A lot of Solr's tests
>> use such a uniqueKey. See
>> solr/core/src/test-files/solr/collection1/conf/schema.xml
>>
>>
>> On Tue, Jul 1, 2014 at 3:20 PM, Alexandre Rafalovitch <ar...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I remember reading somewhere that id field (uniqueKey) must be String.
>> > But I cannot find the definitive confirmation, just that it should be
>> > non-analyzed.
>> >
>> > Can I use a single-valued TrieLongField type, with precision set to 0?
>> > Or am I going to hit issues?
>> >
>> > Regards,
>> > Alex.
>> > Personal website: http://www.outerthoughts.com/
>> > Current project: http://www.solr-start.com/ - Accelerating your Solr
>> > proficiency
>> >
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
Re: Restriction on type of uniqueKey field?
Posted by Michael Della Bitta <mi...@appinions.com>.
Alex, maybe you're thinking of constraints put on shard keys?
Michael Della Bitta
Applications Developer
o: +1 646 532 3062
appinions inc.
“The Science of Influence Marketing”
18 East 41st Street
New York, NY 10017
t: @appinions <https://twitter.com/Appinions> | g+:
plus.google.com/appinions
<https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
w: appinions.com <http://www.appinions.com/>
On Tue, Jul 1, 2014 at 7:05 AM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> No, you definitely can have an int or long uniqueKey. A lot of Solr's tests
> use such a uniqueKey. See
> solr/core/src/test-files/solr/collection1/conf/schema.xml
>
>
> On Tue, Jul 1, 2014 at 3:20 PM, Alexandre Rafalovitch <ar...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I remember reading somewhere that id field (uniqueKey) must be String.
> > But I cannot find the definitive confirmation, just that it should be
> > non-analyzed.
> >
> > Can I use a single-valued TrieLongField type, with precision set to 0?
> > Or am I going to hit issues?
> >
> > Regards,
> > Alex.
> > Personal website: http://www.outerthoughts.com/
> > Current project: http://www.solr-start.com/ - Accelerating your Solr
> > proficiency
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: Restriction on type of uniqueKey field?
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
No, you definitely can have an int or long uniqueKey. A lot of Solr's tests
use such a uniqueKey. See
solr/core/src/test-files/solr/collection1/conf/schema.xml
On Tue, Jul 1, 2014 at 3:20 PM, Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> Hello,
>
> I remember reading somewhere that id field (uniqueKey) must be String.
> But I cannot find the definitive confirmation, just that it should be
> non-analyzed.
>
> Can I use a single-valued TrieLongField type, with precision set to 0?
> Or am I going to hit issues?
>
> Regards,
> Alex.
> Personal website: http://www.outerthoughts.com/
> Current project: http://www.solr-start.com/ - Accelerating your Solr
> proficiency
>
--
Regards,
Shalin Shekhar Mangar.