You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rekha <re...@rediffmail.com> on 2018/09/13 08:07:45 UTC
20180913 - Clarification about Limitation
Hi Solr Team,
I am new to SOLR. I need following clarification from you.
How many documents can be stored in one core? Is there any limit for number of fields per document? How many Core’s can be created in on SOLR? Is there any other limitation is there based on the Disk storage size? I mean some of the database has the 10 GM limit, I have asked like that. Can we use SOLR as a database?
Thanks,Rekha Karthick
Re: 20180913 - Clarification about Limitation
Posted by Andrea Gazzarini <a....@sease.io>.
Hi Rekna,
I think nobody can seriously answer to your questions. The only
"serious" answer to your questions, which doesn't help you, is "it
depends"; specifically, it depends on what is / are your goal / goals,
context and so on.
Is not possible, at least in my opinion, to provide such answers under
an absolute perspective because they shouldn't make so much sense. Of
course, there are some "hard" limits (if you have 10GB of available disk
space, your index cannot be greater than that) but I believe you can
ignore that: it's better if you focus on (and provide us some other
details about) your objectives.
Best,
Andrea
On 13/09/18 10:07, Rekha wrote:
> Hi Solr Team,
> I am new to SOLR. I need following clarification from you.
> How many documents can be stored in one core? Is there any limit for number of fields per document? How many Core’s can be created in on SOLR? Is there any other limitation is there based on the Disk storage size? I mean some of the database has the 10 GM limit, I have asked like that. Can we use SOLR as a database?
> Thanks,Rekha Karthick
Re: 20180913 - Clarification about Limitation
Posted by Rahul Singh <ra...@gmail.com>.
Depends on whether you are using Solr or solrcloud. Solrcloud distributes data into shards so it increases overall capacity.
Rahul Singh
Chief Executive Officer
m 202.905.2818
Anant Corporation
1010 Wisconsin Ave NW, Suite 250
Washington, D.C. 20007
We build and manage digital business technology platforms.
On Sep 13, 2018, 4:50 AM -0400, Rekha <re...@rediffmail.com>, wrote:
> Hi Solr Team,
> I am new to SOLR. I need following clarification from you.
> How many documents can be stored in one core? Is there any limit for number of fields per document? How many Core’s can be created in on SOLR? Is there any other limitation is there based on the Disk storage size? I mean some of the database has the 10 GM limit, I have asked like that. Can we use SOLR as a database?
> Thanks,Rekha Karthick
Re: 20180913 - Clarification about Limitation
Posted by Emir Arnautović <em...@sematext.com>.
Hi,
Here are some thought on how to resolve some of “it depends”: http://www.od-bits.com/2018/01/solrelasticsearch-capacity-planning.html <http://www.od-bits.com/2018/01/solrelasticsearch-capacity-planning.html>
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 13 Sep 2018, at 14:59, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 9/13/2018 2:07 AM, Rekha wrote:
>> Hi Solr Team,
>> I am new to SOLR. I need following clarification from you.
>> How many documents can be stored in one core? Is there any limit for number of fields per document? How many Core’s can be created in on SOLR? Is there any other limitation is there based on the Disk storage size? I mean some of the database has the 10 GM limit, I have asked like that. Can we use SOLR as a database?
>
> You *can* use Solr as a database, but I wouldn't. It's not designed for that role. Actual database software is better for that. If all you need is simple data storage, Solr can handle that, but as soon as you start talking about complex operations like JOIN, a real database is FAR better. Solr is a search engine, and in my opinion, that's what it should be used for.
>
> The only HARD limit that Solr has is actually a Lucene limit. Lucene uses the java "int" type for its internal document ID. Which means that the absolute maximum number of documents in one Solr core is 2147483647. That's a little over two billion. You're likely to have scalability problems long before you reach this number, though. Also, this number includes deleted documents, so it's not a good idea to actually get close to the limit. One rough rule of thumb that sometimes gets used: If you have more than one hundred million documents in a single core, you PROBABLY need to think about re-designing your setup.
>
> Using a sharded index (which SolrCloud can do a lot easier than standalone Solr) removes the two billion document limitation for an index -- by spreading the index across multiple Solr cores.
>
> As for storage, you should have enough disk space available so that your index data can triple in size temporarily. This is not a joke -- that's really the recommendation. The way that Lucene operates requires that you have at least *double* capacity, but there are real world situations in which the index can triple in size.
>
> Running with really big indexes means that you also need a lot of memory. Good performance with Solr requires that the operating system has enough memory to effectively cache the often-used parts of the index.
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#RAM
>
> Thanks,
> Shawn
>
Re: 20180913 - Clarification about Limitation
Posted by Shawn Heisey <ap...@elyograg.org>.
On 9/13/2018 2:07 AM, Rekha wrote:
> Hi Solr Team,
> I am new to SOLR. I need following clarification from you.
> How many documents can be stored in one core? Is there any limit for number of fields per document? How many Core’s can be created in on SOLR? Is there any other limitation is there based on the Disk storage size? I mean some of the database has the 10 GM limit, I have asked like that. Can we use SOLR as a database?
You *can* use Solr as a database, but I wouldn't. It's not designed for
that role. Actual database software is better for that. If all you
need is simple data storage, Solr can handle that, but as soon as you
start talking about complex operations like JOIN, a real database is FAR
better. Solr is a search engine, and in my opinion, that's what it
should be used for.
The only HARD limit that Solr has is actually a Lucene limit. Lucene
uses the java "int" type for its internal document ID. Which means that
the absolute maximum number of documents in one Solr core is
2147483647. That's a little over two billion. You're likely to have
scalability problems long before you reach this number, though. Also,
this number includes deleted documents, so it's not a good idea to
actually get close to the limit. One rough rule of thumb that sometimes
gets used: If you have more than one hundred million documents in a
single core, you PROBABLY need to think about re-designing your setup.
Using a sharded index (which SolrCloud can do a lot easier than
standalone Solr) removes the two billion document limitation for an
index -- by spreading the index across multiple Solr cores.
As for storage, you should have enough disk space available so that your
index data can triple in size temporarily. This is not a joke -- that's
really the recommendation. The way that Lucene operates requires that
you have at least *double* capacity, but there are real world situations
in which the index can triple in size.
Running with really big indexes means that you also need a lot of
memory. Good performance with Solr requires that the operating system
has enough memory to effectively cache the often-used parts of the index.
https://wiki.apache.org/solr/SolrPerformanceProblems#RAM
Thanks,
Shawn