You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jpereira <jp...@gmail.com> on 2017/04/10 15:22:16 UTC
Dynamic schema memory consumption
Hello guys,
I manage a Solr cluster and I am experiencing some problems with dynamic
schemas.
The cluster has 16 nodes and 1500 collections with 12 shards per collection
and 2 replicas per shard. The nodes can be divided in 2 major tiers:
- tier1 is composed of 12 machines with 4 physical cores (8 virtual), 32GB
ram and 4TB ssd; these are used mostly for direct queries and data exports;
- tier2 is composed of 4 machines with 20 physical cores (40 virtual),
128GB and 4TB ssd; these are mostly for aggregation queries (facets)
The problem I am experiencing is that when using dynamic schemas, the Solr
heap size rises dramatically.
I have two tier2 machines (lets call them A and B) running one Solr instance
each with 96GB heap size, with 36 collections totaling 3TB of mainly
fixed-schema (55GB schemaless) data indexed in each machine, and the heap
consumption is on average 60GB (it peaks at around 80GB and drops to around
40GB after a GC run).
On the other tier2 machines (C and D) I was running one Solr instance on
each machine with 32GB heap size and 4 fixed schema collections with about
725GB of data indexed in each machine, which took up about 12GB of heap
size. Recently I added 46 collections to these machines with about 220Gb of
data. In order to do this I was forced to raise the heap size to 64GB and
after indexing everything now the machines have an averaged consumption of
48GB (!!!) (max ~55GB, after GC runs ~37GB)
I also noticed that when indexed fixed schema data the CPU utilization is
also dramatically lower. I have around 100 workers indexing fixed schema
data with and CPU utilization rate of about 10%, while I have only one
worker for schemaless data with a CPU utilization cost of about 20%.
So, I have a two big questions here:
1. Is this dramatic rise in resources consumption when using dynamic fields
"normal"?
2. Is there a way to lower the memory requirements? If so, how?
Thanks for your time!
--
View this message in context: http://lucene.472066.n3.nabble.com/Dynamic-schema-memory-consumption-tp4329184.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Dynamic schema memory consumption
Posted by Dorian Hoxha <do...@gmail.com>.
Also you should change the heap 32GB->30GB so you're guaranteed to get
pointer compression. I think you should have no need to increase it more
than this, since most things have moved to out-of-heap stuff, like
docValues etc.
On Tue, Apr 11, 2017 at 12:07 PM, Dorian Hoxha <do...@gmail.com>
wrote:
> Isn't 18K lucene-indexes (1 for each shard, not counting the replicas) a
> little too much for 3TB of data ?
> Something like 0.167GB for each shard ?
> Isn't that too much overhead (i've mostly worked with es but still lucene
> underneath) ?
>
> Can't you use 1/100 the current number of collections ?
>
>
> On Mon, Apr 10, 2017 at 5:22 PM, jpereira <jp...@gmail.com> wrote:
>
>> Hello guys,
>>
>> I manage a Solr cluster and I am experiencing some problems with dynamic
>> schemas.
>>
>> The cluster has 16 nodes and 1500 collections with 12 shards per
>> collection
>> and 2 replicas per shard. The nodes can be divided in 2 major tiers:
>> - tier1 is composed of 12 machines with 4 physical cores (8 virtual),
>> 32GB
>> ram and 4TB ssd; these are used mostly for direct queries and data
>> exports;
>> - tier2 is composed of 4 machines with 20 physical cores (40 virtual),
>> 128GB and 4TB ssd; these are mostly for aggregation queries (facets)
>>
>> The problem I am experiencing is that when using dynamic schemas, the Solr
>> heap size rises dramatically.
>>
>> I have two tier2 machines (lets call them A and B) running one Solr
>> instance
>> each with 96GB heap size, with 36 collections totaling 3TB of mainly
>> fixed-schema (55GB schemaless) data indexed in each machine, and the heap
>> consumption is on average 60GB (it peaks at around 80GB and drops to
>> around
>> 40GB after a GC run).
>>
>> On the other tier2 machines (C and D) I was running one Solr instance on
>> each machine with 32GB heap size and 4 fixed schema collections with about
>> 725GB of data indexed in each machine, which took up about 12GB of heap
>> size. Recently I added 46 collections to these machines with about 220Gb
>> of
>> data. In order to do this I was forced to raise the heap size to 64GB and
>> after indexing everything now the machines have an averaged consumption of
>> 48GB (!!!) (max ~55GB, after GC runs ~37GB)
>>
>> I also noticed that when indexed fixed schema data the CPU utilization is
>> also dramatically lower. I have around 100 workers indexing fixed schema
>> data with and CPU utilization rate of about 10%, while I have only one
>> worker for schemaless data with a CPU utilization cost of about 20%.
>>
>> So, I have a two big questions here:
>> 1. Is this dramatic rise in resources consumption when using dynamic
>> fields
>> "normal"?
>> 2. Is there a way to lower the memory requirements? If so, how?
>>
>> Thanks for your time!
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble
>> .com/Dynamic-schema-memory-consumption-tp4329184.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
Re: Dynamic schema memory consumption
Posted by Dorian Hoxha <do...@gmail.com>.
Here is a small snippet that I copy pated from Shawn Helsey (who is a core
contributor I think, he's good):
> One thing to note: SolrCloud begins to have performance issues when the
> number of collections in the cloud reaches the low hundreds. It's not
> going to scale very well with a collection per user or per mailbox
> unless there aren't very many users. There are people looking into how
> to scale better, but this hasn't really gone anywhere yet. Here's one
> issue about it, with a lot of very dense comments:
>
> https://issues.apache.org/jira/browse/SOLR-7191
On Tue, Apr 11, 2017 at 9:11 PM, Dorian Hoxha <do...@gmail.com>
wrote:
> And this overhead depends on what? I mean, if I create an empty collection
>> will it take up much heap size just for "being there" ?
>
> Yes. You can search on elastic-search/solr/lucene mailing lists and see
> that it's true. But nobody has `empty` collections, so yours will have a
> schema and some data/segments and translog.
>
> On Tue, Apr 11, 2017 at 7:41 PM, jpereira <jp...@gmail.com> wrote:
>
>> The way the data is spread across the cluster is not really uniform. Most
>> of
>> shards have way lower than 50GB; I would say about 15% of the total shards
>> have more than 50GB.
>>
>>
>> Dorian Hoxha wrote
>> > Each shard is a lucene index which has a lot of overhead.
>>
>> And this overhead depends on what? I mean, if I create an empty collection
>> will it take up much heap size just for "being there" ?
>>
>>
>> Dorian Hoxha wrote
>> > I don't know about static/dynamic memory-issue though.
>>
>> I could not find anything related in the docs or the mailing list either,
>> but I'm still not ready to discard this suspicion...
>>
>> Again, thx for your time
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble
>> .com/Dynamic-schema-memory-consumption-tp4329184p4329367.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
Re: Dynamic schema memory consumption
Posted by Dorian Hoxha <do...@gmail.com>.
>
> And this overhead depends on what? I mean, if I create an empty collection
> will it take up much heap size just for "being there" ?
Yes. You can search on elastic-search/solr/lucene mailing lists and see
that it's true. But nobody has `empty` collections, so yours will have a
schema and some data/segments and translog.
On Tue, Apr 11, 2017 at 7:41 PM, jpereira <jp...@gmail.com> wrote:
> The way the data is spread across the cluster is not really uniform. Most
> of
> shards have way lower than 50GB; I would say about 15% of the total shards
> have more than 50GB.
>
>
> Dorian Hoxha wrote
> > Each shard is a lucene index which has a lot of overhead.
>
> And this overhead depends on what? I mean, if I create an empty collection
> will it take up much heap size just for "being there" ?
>
>
> Dorian Hoxha wrote
> > I don't know about static/dynamic memory-issue though.
>
> I could not find anything related in the docs or the mailing list either,
> but I'm still not ready to discard this suspicion...
>
> Again, thx for your time
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Dynamic-schema-memory-consumption-tp4329184p4329367.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Dynamic schema memory consumption
Posted by jpereira <jp...@gmail.com>.
The way the data is spread across the cluster is not really uniform. Most of
shards have way lower than 50GB; I would say about 15% of the total shards
have more than 50GB.
Dorian Hoxha wrote
> Each shard is a lucene index which has a lot of overhead.
And this overhead depends on what? I mean, if I create an empty collection
will it take up much heap size just for "being there" ?
Dorian Hoxha wrote
> I don't know about static/dynamic memory-issue though.
I could not find anything related in the docs or the mailing list either,
but I'm still not ready to discard this suspicion...
Again, thx for your time
--
View this message in context: http://lucene.472066.n3.nabble.com/Dynamic-schema-memory-consumption-tp4329184p4329367.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Dynamic schema memory consumption
Posted by Dorian Hoxha <do...@gmail.com>.
What I'm suggesting, is that you should aim for max(50GB) per shard of
data. How much is it currently ?
Each shard is a lucene index which has a lot of overhead. If you can, try
to have 20x-50x-100x less shards than you currently do and you'll see lower
heap requirement. I don't know about static/dynamic memory-issue though.
On Tue, Apr 11, 2017 at 6:09 PM, jpereira <jp...@gmail.com> wrote:
> Dorian Hoxha wrote
> > Isn't 18K lucene-indexes (1 for each shard, not counting the replicas) a
> > little too much for 3TB of data ?
> > Something like 0.167GB for each shard ?
> > Isn't that too much overhead (i've mostly worked with es but still lucene
> > underneath) ?
>
> I don't have only 3TB , I have 3TB in two tier2 machines, the whole cluster
> is 12 TB :) So what I was trying to explain was this:
>
> NODES A & B
> 3TB per machine , 36 collections * 12 shards (432 indexes) , average heap
> footprint of 60GB
>
> NODES C & D - at first
> ~725GB per machine, 4 collections * 12 shards (48 indexes) , average heap
> footprint of 12GB
>
> NODES C & D - after addding 220GB schemaless data
> ~1TB per machine, 46 collections * 12 shards (552 indexes), average heap
> footprint of 48GB
>
> So, what you are suggesting is that the culprit for the bump in heap
> footprint is the new collections?
>
>
> Dorian Hoxha wrote
> > Also you should change the heap 32GB->30GB so you're guaranteed to get
> > pointer compression. I think you should have no need to increase it more
> > than this, since most things have moved to out-of-heap stuff, like
> > docValues etc.
>
> I was forced to raise the heap size because the memory requirements
> dramatically raised, hence this post :)
>
> Thanks
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Dynamic-schema-memory-consumption-tp4329184p4329345.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Dynamic schema memory consumption
Posted by jpereira <jp...@gmail.com>.
Dorian Hoxha wrote
> Isn't 18K lucene-indexes (1 for each shard, not counting the replicas) a
> little too much for 3TB of data ?
> Something like 0.167GB for each shard ?
> Isn't that too much overhead (i've mostly worked with es but still lucene
> underneath) ?
I don't have only 3TB , I have 3TB in two tier2 machines, the whole cluster
is 12 TB :) So what I was trying to explain was this:
NODES A & B
3TB per machine , 36 collections * 12 shards (432 indexes) , average heap
footprint of 60GB
NODES C & D - at first
~725GB per machine, 4 collections * 12 shards (48 indexes) , average heap
footprint of 12GB
NODES C & D - after addding 220GB schemaless data
~1TB per machine, 46 collections * 12 shards (552 indexes), average heap
footprint of 48GB
So, what you are suggesting is that the culprit for the bump in heap
footprint is the new collections?
Dorian Hoxha wrote
> Also you should change the heap 32GB->30GB so you're guaranteed to get
> pointer compression. I think you should have no need to increase it more
> than this, since most things have moved to out-of-heap stuff, like
> docValues etc.
I was forced to raise the heap size because the memory requirements
dramatically raised, hence this post :)
Thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/Dynamic-schema-memory-consumption-tp4329184p4329345.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Dynamic schema memory consumption
Posted by Dorian Hoxha <do...@gmail.com>.
Isn't 18K lucene-indexes (1 for each shard, not counting the replicas) a
little too much for 3TB of data ?
Something like 0.167GB for each shard ?
Isn't that too much overhead (i've mostly worked with es but still lucene
underneath) ?
Can't you use 1/100 the current number of collections ?
On Mon, Apr 10, 2017 at 5:22 PM, jpereira <jp...@gmail.com> wrote:
> Hello guys,
>
> I manage a Solr cluster and I am experiencing some problems with dynamic
> schemas.
>
> The cluster has 16 nodes and 1500 collections with 12 shards per collection
> and 2 replicas per shard. The nodes can be divided in 2 major tiers:
> - tier1 is composed of 12 machines with 4 physical cores (8 virtual), 32GB
> ram and 4TB ssd; these are used mostly for direct queries and data exports;
> - tier2 is composed of 4 machines with 20 physical cores (40 virtual),
> 128GB and 4TB ssd; these are mostly for aggregation queries (facets)
>
> The problem I am experiencing is that when using dynamic schemas, the Solr
> heap size rises dramatically.
>
> I have two tier2 machines (lets call them A and B) running one Solr
> instance
> each with 96GB heap size, with 36 collections totaling 3TB of mainly
> fixed-schema (55GB schemaless) data indexed in each machine, and the heap
> consumption is on average 60GB (it peaks at around 80GB and drops to around
> 40GB after a GC run).
>
> On the other tier2 machines (C and D) I was running one Solr instance on
> each machine with 32GB heap size and 4 fixed schema collections with about
> 725GB of data indexed in each machine, which took up about 12GB of heap
> size. Recently I added 46 collections to these machines with about 220Gb of
> data. In order to do this I was forced to raise the heap size to 64GB and
> after indexing everything now the machines have an averaged consumption of
> 48GB (!!!) (max ~55GB, after GC runs ~37GB)
>
> I also noticed that when indexed fixed schema data the CPU utilization is
> also dramatically lower. I have around 100 workers indexing fixed schema
> data with and CPU utilization rate of about 10%, while I have only one
> worker for schemaless data with a CPU utilization cost of about 20%.
>
> So, I have a two big questions here:
> 1. Is this dramatic rise in resources consumption when using dynamic fields
> "normal"?
> 2. Is there a way to lower the memory requirements? If so, how?
>
> Thanks for your time!
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble
> .com/Dynamic-schema-memory-consumption-tp4329184.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>