You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Midas A <te...@gmail.com> on 2019/02/21 15:01:44 UTC
dynamic field issue
Hi All,
How many dynamic field we can create in solr ?. is there any limitation ?
Is indexing dynamic field can increase heap memory on server .
Regards,
Midas
Re: dynamic field issue
Posted by Erick Erickson <er...@gmail.com>.
Dynamic fields are exactly the same as statically defined fields at the Lucene level, so this is exactly equivalent to “would defining N fields statically increase heap”.
IOW, no more than defining that many fields manually.
Best,
Erick
> On Feb 21, 2019, at 8:40 AM, Midas A <te...@gmail.com> wrote:
>
> Here we are indexing dynamic fields and we are using one of this field in*
> bf *.
> Would only indexing dynamic field will increase heap and load of master -
> slave solr servers ?
>
>
> Regards,
> Midas
>
> On Thu, Feb 21, 2019 at 10:03 PM Erick Erickson <er...@gmail.com>
> wrote:
>
>> 300 is still not excessive. As far as memory goes, sure. If you’re
>> faceting, grouping, or sorting docValues would _certainly_ help with memory
>> consumption.
>>
>>> On Feb 21, 2019, at 8:31 AM, Midas A <te...@gmail.com> wrote:
>>>
>>> Hi ,
>>> Plelase help me here we have crossed 100+ fields per dyanmic fields and
>> we
>>> have three dynamic fields.
>>> using docValues in dynamic fields will help in improving heap and query
>>> time ?
>>>
>>> Regards,
>>> Abhishek Tiwari
>>>
>>>
>>> On Thu, Feb 21, 2019 at 9:38 PM Midas A <te...@gmail.com> wrote:
>>>
>>>> Yes . We have crossed 100 fields .
>>>>
>>>> Would docValues help here ?
>>>>
>>>> What kind of information you want from my side ?
>>>>
>>>> On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, <er...@gmail.com>
>>>> wrote:
>>>>
>>>>> There’s no way to answer this given you’ve provided almost no
>>>>> information.
>>>>>
>>>>> Do note that once you get to more than a few hundred fields,
>>>>> Solr still functions, but I’ve seen performance degrade and
>>>>> memory increase.
>>>>>
>>>>> Best,
>>>>> Erick
>>>>>
>>>>>> On Feb 21, 2019, at 7:54 AM, Midas A <te...@gmail.com> wrote:
>>>>>>
>>>>>> Thanks for quick reply .
>>>>>>
>>>>>> we are creating search *query(keyword)* for dynamic field creation
>> to
>>>>>> use click ,cart and order data in search.
>>>>>>
>>>>>> But we are experiencing more heap and increase in query time .
>>>>>> What could be the problem? can be anything related to it ?
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey <ap...@elyograg.org>
>>>>> wrote:
>>>>>>
>>>>>>> On 2/21/2019 8:01 AM, Midas A wrote:
>>>>>>>> How many dynamic field we can create in solr ?. is there any
>>>>> limitation ?
>>>>>>>> Is indexing dynamic field can increase heap memory on server .
>>>>>>>
>>>>>>> At the Lucene level, there is absolutely no difference between a
>>>>>>> standard field and a dynamic field. The difference in Solr is how
>> the
>>>>>>> field is defined, nothing more.
>>>>>>>
>>>>>>> Lucene has no hard limitations on the number of fields you can
>> create,
>>>>>>> but the more you have the larger your index will probably be. Larger
>>>>>>> indexes perform slower than smaller ones and require more resources
>>>>> like
>>>>>>> memory.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Shawn
>>>>>>>
>>>>>
>>>>>
>>
>>
Re: dynamic field issue
Posted by Midas A <te...@gmail.com>.
Here we are indexing dynamic fields and we are using one of this field in*
bf *.
Would only indexing dynamic field will increase heap and load of master -
slave solr servers ?
Regards,
Midas
On Thu, Feb 21, 2019 at 10:03 PM Erick Erickson <er...@gmail.com>
wrote:
> 300 is still not excessive. As far as memory goes, sure. If you’re
> faceting, grouping, or sorting docValues would _certainly_ help with memory
> consumption.
>
> > On Feb 21, 2019, at 8:31 AM, Midas A <te...@gmail.com> wrote:
> >
> > Hi ,
> > Plelase help me here we have crossed 100+ fields per dyanmic fields and
> we
> > have three dynamic fields.
> > using docValues in dynamic fields will help in improving heap and query
> > time ?
> >
> > Regards,
> > Abhishek Tiwari
> >
> >
> > On Thu, Feb 21, 2019 at 9:38 PM Midas A <te...@gmail.com> wrote:
> >
> >> Yes . We have crossed 100 fields .
> >>
> >> Would docValues help here ?
> >>
> >> What kind of information you want from my side ?
> >>
> >> On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, <er...@gmail.com>
> >> wrote:
> >>
> >>> There’s no way to answer this given you’ve provided almost no
> >>> information.
> >>>
> >>> Do note that once you get to more than a few hundred fields,
> >>> Solr still functions, but I’ve seen performance degrade and
> >>> memory increase.
> >>>
> >>> Best,
> >>> Erick
> >>>
> >>>> On Feb 21, 2019, at 7:54 AM, Midas A <te...@gmail.com> wrote:
> >>>>
> >>>> Thanks for quick reply .
> >>>>
> >>>> we are creating search *query(keyword)* for dynamic field creation
> to
> >>>> use click ,cart and order data in search.
> >>>>
> >>>> But we are experiencing more heap and increase in query time .
> >>>> What could be the problem? can be anything related to it ?
> >>>>
> >>>>
> >>>> On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey <ap...@elyograg.org>
> >>> wrote:
> >>>>
> >>>>> On 2/21/2019 8:01 AM, Midas A wrote:
> >>>>>> How many dynamic field we can create in solr ?. is there any
> >>> limitation ?
> >>>>>> Is indexing dynamic field can increase heap memory on server .
> >>>>>
> >>>>> At the Lucene level, there is absolutely no difference between a
> >>>>> standard field and a dynamic field. The difference in Solr is how
> the
> >>>>> field is defined, nothing more.
> >>>>>
> >>>>> Lucene has no hard limitations on the number of fields you can
> create,
> >>>>> but the more you have the larger your index will probably be. Larger
> >>>>> indexes perform slower than smaller ones and require more resources
> >>> like
> >>>>> memory.
> >>>>>
> >>>>> Thanks,
> >>>>> Shawn
> >>>>>
> >>>
> >>>
>
>
Re: dynamic field issue
Posted by Erick Erickson <er...@gmail.com>.
300 is still not excessive. As far as memory goes, sure. If you’re faceting, grouping, or sorting docValues would _certainly_ help with memory consumption.
> On Feb 21, 2019, at 8:31 AM, Midas A <te...@gmail.com> wrote:
>
> Hi ,
> Plelase help me here we have crossed 100+ fields per dyanmic fields and we
> have three dynamic fields.
> using docValues in dynamic fields will help in improving heap and query
> time ?
>
> Regards,
> Abhishek Tiwari
>
>
> On Thu, Feb 21, 2019 at 9:38 PM Midas A <te...@gmail.com> wrote:
>
>> Yes . We have crossed 100 fields .
>>
>> Would docValues help here ?
>>
>> What kind of information you want from my side ?
>>
>> On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, <er...@gmail.com>
>> wrote:
>>
>>> There’s no way to answer this given you’ve provided almost no
>>> information.
>>>
>>> Do note that once you get to more than a few hundred fields,
>>> Solr still functions, but I’ve seen performance degrade and
>>> memory increase.
>>>
>>> Best,
>>> Erick
>>>
>>>> On Feb 21, 2019, at 7:54 AM, Midas A <te...@gmail.com> wrote:
>>>>
>>>> Thanks for quick reply .
>>>>
>>>> we are creating search *query(keyword)* for dynamic field creation to
>>>> use click ,cart and order data in search.
>>>>
>>>> But we are experiencing more heap and increase in query time .
>>>> What could be the problem? can be anything related to it ?
>>>>
>>>>
>>>> On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey <ap...@elyograg.org>
>>> wrote:
>>>>
>>>>> On 2/21/2019 8:01 AM, Midas A wrote:
>>>>>> How many dynamic field we can create in solr ?. is there any
>>> limitation ?
>>>>>> Is indexing dynamic field can increase heap memory on server .
>>>>>
>>>>> At the Lucene level, there is absolutely no difference between a
>>>>> standard field and a dynamic field. The difference in Solr is how the
>>>>> field is defined, nothing more.
>>>>>
>>>>> Lucene has no hard limitations on the number of fields you can create,
>>>>> but the more you have the larger your index will probably be. Larger
>>>>> indexes perform slower than smaller ones and require more resources
>>> like
>>>>> memory.
>>>>>
>>>>> Thanks,
>>>>> Shawn
>>>>>
>>>
>>>
Re: dynamic field issue
Posted by Midas A <te...@gmail.com>.
Hi ,
Plelase help me here we have crossed 100+ fields per dyanmic fields and we
have three dynamic fields.
using docValues in dynamic fields will help in improving heap and query
time ?
Regards,
Abhishek Tiwari
On Thu, Feb 21, 2019 at 9:38 PM Midas A <te...@gmail.com> wrote:
> Yes . We have crossed 100 fields .
>
> Would docValues help here ?
>
> What kind of information you want from my side ?
>
> On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, <er...@gmail.com>
> wrote:
>
>> There’s no way to answer this given you’ve provided almost no
>> information.
>>
>> Do note that once you get to more than a few hundred fields,
>> Solr still functions, but I’ve seen performance degrade and
>> memory increase.
>>
>> Best,
>> Erick
>>
>> > On Feb 21, 2019, at 7:54 AM, Midas A <te...@gmail.com> wrote:
>> >
>> > Thanks for quick reply .
>> >
>> > we are creating search *query(keyword)* for dynamic field creation to
>> > use click ,cart and order data in search.
>> >
>> > But we are experiencing more heap and increase in query time .
>> > What could be the problem? can be anything related to it ?
>> >
>> >
>> > On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey <ap...@elyograg.org>
>> wrote:
>> >
>> >> On 2/21/2019 8:01 AM, Midas A wrote:
>> >>> How many dynamic field we can create in solr ?. is there any
>> limitation ?
>> >>> Is indexing dynamic field can increase heap memory on server .
>> >>
>> >> At the Lucene level, there is absolutely no difference between a
>> >> standard field and a dynamic field. The difference in Solr is how the
>> >> field is defined, nothing more.
>> >>
>> >> Lucene has no hard limitations on the number of fields you can create,
>> >> but the more you have the larger your index will probably be. Larger
>> >> indexes perform slower than smaller ones and require more resources
>> like
>> >> memory.
>> >>
>> >> Thanks,
>> >> Shawn
>> >>
>>
>>
Re: dynamic field issue
Posted by Midas A <te...@gmail.com>.
Yes . We have crossed 100 fields .
Would docValues help here ?
What kind of information you want from my side ?
On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, <er...@gmail.com>
wrote:
> There’s no way to answer this given you’ve provided almost no
> information.
>
> Do note that once you get to more than a few hundred fields,
> Solr still functions, but I’ve seen performance degrade and
> memory increase.
>
> Best,
> Erick
>
> > On Feb 21, 2019, at 7:54 AM, Midas A <te...@gmail.com> wrote:
> >
> > Thanks for quick reply .
> >
> > we are creating search *query(keyword)* for dynamic field creation to
> > use click ,cart and order data in search.
> >
> > But we are experiencing more heap and increase in query time .
> > What could be the problem? can be anything related to it ?
> >
> >
> > On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey <ap...@elyograg.org>
> wrote:
> >
> >> On 2/21/2019 8:01 AM, Midas A wrote:
> >>> How many dynamic field we can create in solr ?. is there any
> limitation ?
> >>> Is indexing dynamic field can increase heap memory on server .
> >>
> >> At the Lucene level, there is absolutely no difference between a
> >> standard field and a dynamic field. The difference in Solr is how the
> >> field is defined, nothing more.
> >>
> >> Lucene has no hard limitations on the number of fields you can create,
> >> but the more you have the larger your index will probably be. Larger
> >> indexes perform slower than smaller ones and require more resources like
> >> memory.
> >>
> >> Thanks,
> >> Shawn
> >>
>
>
Re: dynamic field issue
Posted by Erick Erickson <er...@gmail.com>.
There’s no way to answer this given you’ve provided almost no
information.
Do note that once you get to more than a few hundred fields,
Solr still functions, but I’ve seen performance degrade and
memory increase.
Best,
Erick
> On Feb 21, 2019, at 7:54 AM, Midas A <te...@gmail.com> wrote:
>
> Thanks for quick reply .
>
> we are creating search *query(keyword)* for dynamic field creation to
> use click ,cart and order data in search.
>
> But we are experiencing more heap and increase in query time .
> What could be the problem? can be anything related to it ?
>
>
> On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 2/21/2019 8:01 AM, Midas A wrote:
>>> How many dynamic field we can create in solr ?. is there any limitation ?
>>> Is indexing dynamic field can increase heap memory on server .
>>
>> At the Lucene level, there is absolutely no difference between a
>> standard field and a dynamic field. The difference in Solr is how the
>> field is defined, nothing more.
>>
>> Lucene has no hard limitations on the number of fields you can create,
>> but the more you have the larger your index will probably be. Larger
>> indexes perform slower than smaller ones and require more resources like
>> memory.
>>
>> Thanks,
>> Shawn
>>
Re: dynamic field issue
Posted by Midas A <te...@gmail.com>.
Thanks for quick reply .
we are creating search *query(keyword)* for dynamic field creation to
use click ,cart and order data in search.
But we are experiencing more heap and increase in query time .
What could be the problem? can be anything related to it ?
On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/21/2019 8:01 AM, Midas A wrote:
> > How many dynamic field we can create in solr ?. is there any limitation ?
> > Is indexing dynamic field can increase heap memory on server .
>
> At the Lucene level, there is absolutely no difference between a
> standard field and a dynamic field. The difference in Solr is how the
> field is defined, nothing more.
>
> Lucene has no hard limitations on the number of fields you can create,
> but the more you have the larger your index will probably be. Larger
> indexes perform slower than smaller ones and require more resources like
> memory.
>
> Thanks,
> Shawn
>
Re: dynamic field issue
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/21/2019 8:01 AM, Midas A wrote:
> How many dynamic field we can create in solr ?. is there any limitation ?
> Is indexing dynamic field can increase heap memory on server .
At the Lucene level, there is absolutely no difference between a
standard field and a dynamic field. The difference in Solr is how the
field is defined, nothing more.
Lucene has no hard limitations on the number of fields you can create,
but the more you have the larger your index will probably be. Larger
indexes perform slower than smaller ones and require more resources like
memory.
Thanks,
Shawn