You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Macroman <pe...@hotmail.com> on 2013/02/07 12:19:53 UTC

Maximum Number of Records In Index

I have searched this forum but not yet found a definitive answer, I think the
answer is "There is No Limit depends on server specification". But never the
less I will say what I have seen and then ask the questions.

>From scratch (November 2011) I have set up our SOLR which contains data from
various sources, since March 2012 , the number of indexed records (unique
ID's) reached 13.5 million , which was to be expected. However for the last
8 months the number of records in the index has not gone above 13.5 million,
yet looking at the request handler outputs I can safely say at least
anywhere from 50 thousand to 100 thousand records are being indexed daily.
So I am assuming that earlier records are being removed, and I do not want
that.

Question: If there is a limit to the number of records the index can store
where do I find this and change it?
Question: If there is no limit does anyone have any idea why for the last
months the number has not gone beyond 13.5 million, I can safely say that at
least 90% are new records.

thanks

macroman



--
View this message in context: http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Maximum Number of Records In Index

Posted by Otis Gospodnetic <ot...@gmail.com>.
We don't run one ourselves at Sematext, but know of people who do have
large ES clusters, one with > 10B docs.

Otis
--
Solr & ElasticSearch Support
http://sematext.com/





On Mon, Feb 11, 2013 at 8:41 AM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:

> Otis,
> Do you run 4bn docs SolrCloud or ElasticSearch or aware of somebody who do?
> 10.02.2013 4:54 пользователь "Otis Gospodnetic" <
> otis.gospodnetic@gmail.com>
> написал:
>
> > Exceeding 2B is no problem. But it won't happen in a single Lucene index
> > any time soon,  so...
> >
> > Otis
> > Solr & ElasticSearch Support
> > http://sematext.com/
> > On Feb 7, 2013 10:08 AM, "Mikhail Khludnev" <mk...@griddynamics.com>
> > wrote:
> >
> > > Actually, I have a dream to exceed those two billions. It seems
> possible,
> > > to move to Vint in fileformat and change int docnums to longs in Lucene
> > > API. Does anyone know whether it's possible?
> > > And this question is not so esoteric if we are talking about SolrCloud,
> > > which can hold more that 2bn docs in few smaller shards. Any
> experience?
> > >
> > >
> > > On Thu, Feb 7, 2013 at 5:46 PM, Rafał Kuć <r....@solr.pl> wrote:
> > >
> > > > Hello!
> > > >
> > > > Right, my bad - ids are still using int32. However, that still
> > > > gives us 2,147,483,648 possible identifiers right per single index,
> > > > which is not close to the 13,5 millions mentioned in the first mail.
> > > >
> > > > --
> > > > Regards,
> > > >  Rafał Kuć
> > > >  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
> > > ElasticSearch
> > > >
> > > > > Rafal,
> > > >
> > > > > What about docnums, don't they are limited by int32 ?
> > > > > 07.02.2013 15:33 пользователь "Rafał Kuć" <r....@solr.pl> написал:
> > > >
> > > > >> Hello!
> > > > >>
> > > > >> Practically there is no limit in how many documents can be stored
> > in a
> > > > >> single index. In your case, as you are using Solr from 2011, there
> > is
> > > > >> a limitation regarding the number of unique terms per Lucene
> segment
> > > > >> (
> > > > >>
> > > >
> > >
> >
> http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations
> > > > >> ).
> > > > >> However I don't think you've hit that. Solr by itself doesn't
> remove
> > > > >> documents unless told to do so.
> > > > >>
> > > > >> Its hard to guess what can be the reason and as you said, you see
> > > > >> updates coming to your handler. Maybe new documents have the same
> > > > >> identifiers that the ones that are already indexed ? As I said,
> this
> > > > >> is only a guess and we would need to have more information. Are
> > there
> > > > >> any exceptions in the logs ? Do you run delete command ? Are your
> > > > >> index files changed ? How do you run commit ?
> > > > >>
> > > > >> --
> > > > >> Regards,
> > > > >>  Rafał Kuć
> > > > >>  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
> > > > ElasticSearch
> > > > >>
> > > > >> > I have searched this forum but not yet found a definitive
> answer,
> > I
> > > > >> think the
> > > > >> > answer is "There is No Limit depends on server specification".
> But
> > > > never
> > > > >> the
> > > > >> > less I will say what I have seen and then ask the questions.
> > > > >>
> > > > >> > From scratch (November 2011) I have set up our SOLR which
> contains
> > > > data
> > > > >> from
> > > > >> > various sources, since March 2012 , the number of indexed
> records
> > > > (unique
> > > > >> > ID's) reached 13.5 million , which was to be expected. However
> for
> > > the
> > > > >> last
> > > > >> > 8 months the number of records in the index has not gone above
> > 13.5
> > > > >> million,
> > > > >> > yet looking at the request handler outputs I can safely say at
> > least
> > > > >> > anywhere from 50 thousand to 100 thousand records are being
> > indexed
> > > > >> daily.
> > > > >> > So I am assuming that earlier records are being removed, and I
> do
> > > not
> > > > >> want
> > > > >> > that.
> > > > >>
> > > > >> > Question: If there is a limit to the number of records the index
> > can
> > > > >> store
> > > > >> > where do I find this and change it?
> > > > >> > Question: If there is no limit does anyone have any idea why for
> > the
> > > > last
> > > > >> > months the number has not gone beyond 13.5 million, I can safely
> > say
> > > > >> that at
> > > > >> > least 90% are new records.
> > > > >>
> > > > >> > thanks
> > > > >>
> > > > >> > macroman
> > > > >>
> > > > >>
> > > > >>
> > > > >> > --
> > > > >> > View this message in context:
> > > > >> >
> > > > >>
> > > >
> > >
> >
> http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html
> > > > >> > Sent from the Solr - User mailing list archive at Nabble.com.
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > > Principal Engineer,
> > > Grid Dynamics
> > >
> > > <http://www.griddynamics.com>
> > >  <mk...@griddynamics.com>
> > >
> >
>

Re: Maximum Number of Records In Index

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Otis,
Do you run 4bn docs SolrCloud or ElasticSearch or aware of somebody who do?
10.02.2013 4:54 пользователь "Otis Gospodnetic" <ot...@gmail.com>
написал:

> Exceeding 2B is no problem. But it won't happen in a single Lucene index
> any time soon,  so...
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
> On Feb 7, 2013 10:08 AM, "Mikhail Khludnev" <mk...@griddynamics.com>
> wrote:
>
> > Actually, I have a dream to exceed those two billions. It seems possible,
> > to move to Vint in fileformat and change int docnums to longs in Lucene
> > API. Does anyone know whether it's possible?
> > And this question is not so esoteric if we are talking about SolrCloud,
> > which can hold more that 2bn docs in few smaller shards. Any experience?
> >
> >
> > On Thu, Feb 7, 2013 at 5:46 PM, Rafał Kuć <r....@solr.pl> wrote:
> >
> > > Hello!
> > >
> > > Right, my bad - ids are still using int32. However, that still
> > > gives us 2,147,483,648 possible identifiers right per single index,
> > > which is not close to the 13,5 millions mentioned in the first mail.
> > >
> > > --
> > > Regards,
> > >  Rafał Kuć
> > >  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
> > ElasticSearch
> > >
> > > > Rafal,
> > >
> > > > What about docnums, don't they are limited by int32 ?
> > > > 07.02.2013 15:33 пользователь "Rafał Kuć" <r....@solr.pl> написал:
> > >
> > > >> Hello!
> > > >>
> > > >> Practically there is no limit in how many documents can be stored
> in a
> > > >> single index. In your case, as you are using Solr from 2011, there
> is
> > > >> a limitation regarding the number of unique terms per Lucene segment
> > > >> (
> > > >>
> > >
> >
> http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations
> > > >> ).
> > > >> However I don't think you've hit that. Solr by itself doesn't remove
> > > >> documents unless told to do so.
> > > >>
> > > >> Its hard to guess what can be the reason and as you said, you see
> > > >> updates coming to your handler. Maybe new documents have the same
> > > >> identifiers that the ones that are already indexed ? As I said, this
> > > >> is only a guess and we would need to have more information. Are
> there
> > > >> any exceptions in the logs ? Do you run delete command ? Are your
> > > >> index files changed ? How do you run commit ?
> > > >>
> > > >> --
> > > >> Regards,
> > > >>  Rafał Kuć
> > > >>  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
> > > ElasticSearch
> > > >>
> > > >> > I have searched this forum but not yet found a definitive answer,
> I
> > > >> think the
> > > >> > answer is "There is No Limit depends on server specification". But
> > > never
> > > >> the
> > > >> > less I will say what I have seen and then ask the questions.
> > > >>
> > > >> > From scratch (November 2011) I have set up our SOLR which contains
> > > data
> > > >> from
> > > >> > various sources, since March 2012 , the number of indexed records
> > > (unique
> > > >> > ID's) reached 13.5 million , which was to be expected. However for
> > the
> > > >> last
> > > >> > 8 months the number of records in the index has not gone above
> 13.5
> > > >> million,
> > > >> > yet looking at the request handler outputs I can safely say at
> least
> > > >> > anywhere from 50 thousand to 100 thousand records are being
> indexed
> > > >> daily.
> > > >> > So I am assuming that earlier records are being removed, and I do
> > not
> > > >> want
> > > >> > that.
> > > >>
> > > >> > Question: If there is a limit to the number of records the index
> can
> > > >> store
> > > >> > where do I find this and change it?
> > > >> > Question: If there is no limit does anyone have any idea why for
> the
> > > last
> > > >> > months the number has not gone beyond 13.5 million, I can safely
> say
> > > >> that at
> > > >> > least 90% are new records.
> > > >>
> > > >> > thanks
> > > >>
> > > >> > macroman
> > > >>
> > > >>
> > > >>
> > > >> > --
> > > >> > View this message in context:
> > > >> >
> > > >>
> > >
> >
> http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html
> > > >> > Sent from the Solr - User mailing list archive at Nabble.com.
> > > >>
> > > >>
> > >
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> >  <mk...@griddynamics.com>
> >
>

Re: Maximum Number of Records In Index

Posted by Otis Gospodnetic <ot...@gmail.com>.
Exceeding 2B is no problem. But it won't happen in a single Lucene index
any time soon,  so...

Otis
Solr & ElasticSearch Support
http://sematext.com/
On Feb 7, 2013 10:08 AM, "Mikhail Khludnev" <mk...@griddynamics.com>
wrote:

> Actually, I have a dream to exceed those two billions. It seems possible,
> to move to Vint in fileformat and change int docnums to longs in Lucene
> API. Does anyone know whether it's possible?
> And this question is not so esoteric if we are talking about SolrCloud,
> which can hold more that 2bn docs in few smaller shards. Any experience?
>
>
> On Thu, Feb 7, 2013 at 5:46 PM, Rafał Kuć <r....@solr.pl> wrote:
>
> > Hello!
> >
> > Right, my bad - ids are still using int32. However, that still
> > gives us 2,147,483,648 possible identifiers right per single index,
> > which is not close to the 13,5 millions mentioned in the first mail.
> >
> > --
> > Regards,
> >  Rafał Kuć
> >  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
> ElasticSearch
> >
> > > Rafal,
> >
> > > What about docnums, don't they are limited by int32 ?
> > > 07.02.2013 15:33 пользователь "Rafał Kuć" <r....@solr.pl> написал:
> >
> > >> Hello!
> > >>
> > >> Practically there is no limit in how many documents can be stored in a
> > >> single index. In your case, as you are using Solr from 2011, there is
> > >> a limitation regarding the number of unique terms per Lucene segment
> > >> (
> > >>
> >
> http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations
> > >> ).
> > >> However I don't think you've hit that. Solr by itself doesn't remove
> > >> documents unless told to do so.
> > >>
> > >> Its hard to guess what can be the reason and as you said, you see
> > >> updates coming to your handler. Maybe new documents have the same
> > >> identifiers that the ones that are already indexed ? As I said, this
> > >> is only a guess and we would need to have more information. Are there
> > >> any exceptions in the logs ? Do you run delete command ? Are your
> > >> index files changed ? How do you run commit ?
> > >>
> > >> --
> > >> Regards,
> > >>  Rafał Kuć
> > >>  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
> > ElasticSearch
> > >>
> > >> > I have searched this forum but not yet found a definitive answer, I
> > >> think the
> > >> > answer is "There is No Limit depends on server specification". But
> > never
> > >> the
> > >> > less I will say what I have seen and then ask the questions.
> > >>
> > >> > From scratch (November 2011) I have set up our SOLR which contains
> > data
> > >> from
> > >> > various sources, since March 2012 , the number of indexed records
> > (unique
> > >> > ID's) reached 13.5 million , which was to be expected. However for
> the
> > >> last
> > >> > 8 months the number of records in the index has not gone above 13.5
> > >> million,
> > >> > yet looking at the request handler outputs I can safely say at least
> > >> > anywhere from 50 thousand to 100 thousand records are being indexed
> > >> daily.
> > >> > So I am assuming that earlier records are being removed, and I do
> not
> > >> want
> > >> > that.
> > >>
> > >> > Question: If there is a limit to the number of records the index can
> > >> store
> > >> > where do I find this and change it?
> > >> > Question: If there is no limit does anyone have any idea why for the
> > last
> > >> > months the number has not gone beyond 13.5 million, I can safely say
> > >> that at
> > >> > least 90% are new records.
> > >>
> > >> > thanks
> > >>
> > >> > macroman
> > >>
> > >>
> > >>
> > >> > --
> > >> > View this message in context:
> > >> >
> > >>
> >
> http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html
> > >> > Sent from the Solr - User mailing list archive at Nabble.com.
> > >>
> > >>
> >
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
>  <mk...@griddynamics.com>
>

Re: Maximum Number of Records In Index

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Actually, I have a dream to exceed those two billions. It seems possible,
to move to Vint in fileformat and change int docnums to longs in Lucene
API. Does anyone know whether it's possible?
And this question is not so esoteric if we are talking about SolrCloud,
which can hold more that 2bn docs in few smaller shards. Any experience?


On Thu, Feb 7, 2013 at 5:46 PM, Rafał Kuć <r....@solr.pl> wrote:

> Hello!
>
> Right, my bad - ids are still using int32. However, that still
> gives us 2,147,483,648 possible identifiers right per single index,
> which is not close to the 13,5 millions mentioned in the first mail.
>
> --
> Regards,
>  Rafał Kuć
>  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
>
> > Rafal,
>
> > What about docnums, don't they are limited by int32 ?
> > 07.02.2013 15:33 пользователь "Rafał Kuć" <r....@solr.pl> написал:
>
> >> Hello!
> >>
> >> Practically there is no limit in how many documents can be stored in a
> >> single index. In your case, as you are using Solr from 2011, there is
> >> a limitation regarding the number of unique terms per Lucene segment
> >> (
> >>
> http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations
> >> ).
> >> However I don't think you've hit that. Solr by itself doesn't remove
> >> documents unless told to do so.
> >>
> >> Its hard to guess what can be the reason and as you said, you see
> >> updates coming to your handler. Maybe new documents have the same
> >> identifiers that the ones that are already indexed ? As I said, this
> >> is only a guess and we would need to have more information. Are there
> >> any exceptions in the logs ? Do you run delete command ? Are your
> >> index files changed ? How do you run commit ?
> >>
> >> --
> >> Regards,
> >>  Rafał Kuć
> >>  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
> ElasticSearch
> >>
> >> > I have searched this forum but not yet found a definitive answer, I
> >> think the
> >> > answer is "There is No Limit depends on server specification". But
> never
> >> the
> >> > less I will say what I have seen and then ask the questions.
> >>
> >> > From scratch (November 2011) I have set up our SOLR which contains
> data
> >> from
> >> > various sources, since March 2012 , the number of indexed records
> (unique
> >> > ID's) reached 13.5 million , which was to be expected. However for the
> >> last
> >> > 8 months the number of records in the index has not gone above 13.5
> >> million,
> >> > yet looking at the request handler outputs I can safely say at least
> >> > anywhere from 50 thousand to 100 thousand records are being indexed
> >> daily.
> >> > So I am assuming that earlier records are being removed, and I do not
> >> want
> >> > that.
> >>
> >> > Question: If there is a limit to the number of records the index can
> >> store
> >> > where do I find this and change it?
> >> > Question: If there is no limit does anyone have any idea why for the
> last
> >> > months the number has not gone beyond 13.5 million, I can safely say
> >> that at
> >> > least 90% are new records.
> >>
> >> > thanks
> >>
> >> > macroman
> >>
> >>
> >>
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html
> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
>
>


-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
 <mk...@griddynamics.com>

Re: Maximum Number of Records In Index

Posted by Rafał Kuć <r....@solr.pl>.
Hello!

Right, my bad - ids are still using int32. However, that still
gives us 2,147,483,648 possible identifiers right per single index,
which is not close to the 13,5 millions mentioned in the first mail.

-- 
Regards,
 Rafał Kuć
 Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch

> Rafal,

> What about docnums, don't they are limited by int32 ?
> 07.02.2013 15:33 пользователь "Rafał Kuć" <r....@solr.pl> написал:

>> Hello!
>>
>> Practically there is no limit in how many documents can be stored in a
>> single index. In your case, as you are using Solr from 2011, there is
>> a limitation regarding the number of unique terms per Lucene segment
>> (
>> http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations
>> ).
>> However I don't think you've hit that. Solr by itself doesn't remove
>> documents unless told to do so.
>>
>> Its hard to guess what can be the reason and as you said, you see
>> updates coming to your handler. Maybe new documents have the same
>> identifiers that the ones that are already indexed ? As I said, this
>> is only a guess and we would need to have more information. Are there
>> any exceptions in the logs ? Do you run delete command ? Are your
>> index files changed ? How do you run commit ?
>>
>> --
>> Regards,
>>  Rafał Kuć
>>  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
>>
>> > I have searched this forum but not yet found a definitive answer, I
>> think the
>> > answer is "There is No Limit depends on server specification". But never
>> the
>> > less I will say what I have seen and then ask the questions.
>>
>> > From scratch (November 2011) I have set up our SOLR which contains data
>> from
>> > various sources, since March 2012 , the number of indexed records (unique
>> > ID's) reached 13.5 million , which was to be expected. However for the
>> last
>> > 8 months the number of records in the index has not gone above 13.5
>> million,
>> > yet looking at the request handler outputs I can safely say at least
>> > anywhere from 50 thousand to 100 thousand records are being indexed
>> daily.
>> > So I am assuming that earlier records are being removed, and I do not
>> want
>> > that.
>>
>> > Question: If there is a limit to the number of records the index can
>> store
>> > where do I find this and change it?
>> > Question: If there is no limit does anyone have any idea why for the last
>> > months the number has not gone beyond 13.5 million, I can safely say
>> that at
>> > least 90% are new records.
>>
>> > thanks
>>
>> > macroman
>>
>>
>>
>> > --
>> > View this message in context:
>> >
>> http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>

Re: Maximum Number of Records In Index

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Rafal,

What about docnums, don't they are limited by int32 ?
07.02.2013 15:33 пользователь "Rafał Kuć" <r....@solr.pl> написал:

> Hello!
>
> Practically there is no limit in how many documents can be stored in a
> single index. In your case, as you are using Solr from 2011, there is
> a limitation regarding the number of unique terms per Lucene segment
> (
> http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations
> ).
> However I don't think you've hit that. Solr by itself doesn't remove
> documents unless told to do so.
>
> Its hard to guess what can be the reason and as you said, you see
> updates coming to your handler. Maybe new documents have the same
> identifiers that the ones that are already indexed ? As I said, this
> is only a guess and we would need to have more information. Are there
> any exceptions in the logs ? Do you run delete command ? Are your
> index files changed ? How do you run commit ?
>
> --
> Regards,
>  Rafał Kuć
>  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
>
> > I have searched this forum but not yet found a definitive answer, I
> think the
> > answer is "There is No Limit depends on server specification". But never
> the
> > less I will say what I have seen and then ask the questions.
>
> > From scratch (November 2011) I have set up our SOLR which contains data
> from
> > various sources, since March 2012 , the number of indexed records (unique
> > ID's) reached 13.5 million , which was to be expected. However for the
> last
> > 8 months the number of records in the index has not gone above 13.5
> million,
> > yet looking at the request handler outputs I can safely say at least
> > anywhere from 50 thousand to 100 thousand records are being indexed
> daily.
> > So I am assuming that earlier records are being removed, and I do not
> want
> > that.
>
> > Question: If there is a limit to the number of records the index can
> store
> > where do I find this and change it?
> > Question: If there is no limit does anyone have any idea why for the last
> > months the number has not gone beyond 13.5 million, I can safely say
> that at
> > least 90% are new records.
>
> > thanks
>
> > macroman
>
>
>
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Maximum Number of Records In Index

Posted by Macroman <pe...@hotmail.com>.
Partial updates is nothing as clever as I may have made it sound, it is just
changing a record value , for example last name from Smith to Jones, that's
my partial update. 

No errors at all in indexing, I have not yet checked the logs , but the DIH
output counts show no errors, here is an example
<str name="Total Requests made to DataSource">2</str><str name="Total Rows
Fetched">14823</str><str name="Total Documents Skipped">0</str><str
name="Full Dump Started">2013-02-14 07:00:30</str><str name="">Indexing
completed. Added/Updated: 14823 documents. Deleted 0 documents.</str><str
name="Committed">2013-02-14 07:19:59</str><str name="Optimized">2013-02-14
07:19:59</str><str name="Total Documents Processed">14823</str><str
name="Time taken ">0:19:58.557</str>

Having analysed the SOLR index this afternoon I realised that I actually add
the date/time of when record indexed so did a quick SOLR admin count using
.... record_date:[2000-02-14T00:00:00.000Z TO 2013-02-10T00:00:00.000Z]....
this resulted in a count of 32.723 records indexed today, and when I add up
all the DIH's of Added/Updated it comes to 35,369 , weird !!! Now for the
total maths , yesterday's total index count was 13593885 and today it is
13598211 a difference of 4326, but I do need to take into account records
updates, so running the SQL form each of the DIH's sources in SQL Developer
to purely get counts, my counts are a total of 31,789 which means only 3,000
to 4,000 updates the rest are all new.

So I will definately say that records are being deleted so need to check the
logs as suggested. If no mention of deletions exist my next question will be
can I get a Month-YYYY breakdown on a SOLR date field so I can monitor
records that drop off, because one field that will definately not change is
the record creation date from the source systems which is part of the
indexed record?

<this line ready for entering log details to see if any deletes occurred>



--
View this message in context: http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961p4040445.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Maximum Number of Records In Index

Posted by Joel Bernstein <jo...@gmail.com>.
A couple of things to check.

1) Have you retained your solr logs. If so, take a look in them for
indexing errors.
2) What is the difference between maxdocs and numdocs. This will give an
indication if a large number of records are being deleted or updated.
3) Can you explain your partial updates? Are you sending the entire
document again for the partial update?


Try to debug your next load. Perform the load, and watch the logs for
errors. Write a program that loops through each doc and checks to see if
the doc is present in the index.





On Tue, Feb 12, 2013 at 6:19 AM, Macroman <pe...@hotmail.com> wrote:

> Our document ID's are most definately distinct and there are partial
> updates
> to existing records, I have run SQL queries outside of SOLR to validate
> records going in and only about 1% are updates to existing records. There
> are no deletes underway every day new records are added or updated. Example
> for today. Before Data Handler ran, 13,586,537 records in SOLR all distinct
> ID's. Records extracted from 7 different sources to go into SOLR index was
> ,
> 45,345, of these 1,912 were updates to existing records. Thus 43,433 were
> new records each with a new ID. I made sure ID's we always distinct. Yet
> our
> index now says 13,589,646. Indicating that only 3,109 new records went into
> the index. Thus missing 40,324 records. I use Date Facet Range and can see
> that there is an increase for January and February this year. In conclusion
> I have to say that it must be removing earlier records somehow despite no
> knowing where this may be controlled/set if at all. If there is a possible
> configuration to remove or weed records where is this configured? Our SOLR
> is virtually out of the box and only SOLCONFIG and SCHEMA amended to suit
> the needs of our business for fields and field types indexed. We also have
> configured the macro.s "Velocity" to display results. So none the wiser and
> thank you to all whom have responded so far.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961p4039908.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Joel Bernstein
Professional Services LucidWorks

Re: Maximum Number of Records In Index

Posted by Macroman <pe...@hotmail.com>.
Our document ID's are most definately distinct and there are partial updates
to existing records, I have run SQL queries outside of SOLR to validate
records going in and only about 1% are updates to existing records. There
are no deletes underway every day new records are added or updated. Example
for today. Before Data Handler ran, 13,586,537 records in SOLR all distinct
ID's. Records extracted from 7 different sources to go into SOLR index was ,
45,345, of these 1,912 were updates to existing records. Thus 43,433 were
new records each with a new ID. I made sure ID's we always distinct. Yet our
index now says 13,589,646. Indicating that only 3,109 new records went into
the index. Thus missing 40,324 records. I use Date Facet Range and can see
that there is an increase for January and February this year. In conclusion
I have to say that it must be removing earlier records somehow despite no
knowing where this may be controlled/set if at all. If there is a possible
configuration to remove or weed records where is this configured? Our SOLR
is virtually out of the box and only SOLCONFIG and SCHEMA amended to suit
the needs of our business for fields and field types indexed. We also have
configured the macro.s "Velocity" to display results. So none the wiser and
thank you to all whom have responded so far.



--
View this message in context: http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961p4039908.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Maximum Number of Records In Index

Posted by Rafał Kuć <r....@solr.pl>.
Hello!

Practically there is no limit in how many documents can be stored in a
single index. In your case, as you are using Solr from 2011, there is
a limitation regarding the number of unique terms per Lucene segment
(http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations).
However I don't think you've hit that. Solr by itself doesn't remove
documents unless told to do so.

Its hard to guess what can be the reason and as you said, you see
updates coming to your handler. Maybe new documents have the same
identifiers that the ones that are already indexed ? As I said, this
is only a guess and we would need to have more information. Are there
any exceptions in the logs ? Do you run delete command ? Are your
index files changed ? How do you run commit ?

-- 
Regards,
 Rafał Kuć
 Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch

> I have searched this forum but not yet found a definitive answer, I think the
> answer is "There is No Limit depends on server specification". But never the
> less I will say what I have seen and then ask the questions.

> From scratch (November 2011) I have set up our SOLR which contains data from
> various sources, since March 2012 , the number of indexed records (unique
> ID's) reached 13.5 million , which was to be expected. However for the last
> 8 months the number of records in the index has not gone above 13.5 million,
> yet looking at the request handler outputs I can safely say at least
> anywhere from 50 thousand to 100 thousand records are being indexed daily.
> So I am assuming that earlier records are being removed, and I do not want
> that.

> Question: If there is a limit to the number of records the index can store
> where do I find this and change it?
> Question: If there is no limit does anyone have any idea why for the last
> months the number has not gone beyond 13.5 million, I can safely say that at
> least 90% are new records.

> thanks

> macroman



> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html
> Sent from the Solr - User mailing list archive at Nabble.com.