You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by 韦震宇 <we...@win-trust.com> on 2012/09/25 04:21:23 UTC
How can I create about 100000 independent indexes in Solr?
Dear all,
The company I'm working in have a website to server more than 100000 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer.
The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes.
I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution.
The other solution "Flattening data into a single index" is a choice, but i think it's best to keep all indexes indepent.
Could you tell me how to create about 100000 independent indexes in Solr?
Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Posted by 韦震宇 <we...@win-trust.com>.
Hi, Tanguy
Oh, I understand now. I don't have the issue as you. Though there
are so many customers in our site, but the fields they owned are same.
so few field fields are ok in my scene.
Best Regards!
Monton
----- Original Message -----
From: "Tanguy Moal" <ta...@gmail.com>
To: <so...@lucene.apache.org>
Sent: Thursday, September 27, 2012 4:34 PM
Subject: Re: How can I create about 100000 independent indexes in Solr?
Hello Monton,
I wanted to make sure that you understood me well : I really don't how well
does solr scale if the number of fields increases...
What I mean here is that the more distinct fields you index, the more
memory you will need.
So if in your schema, you have something like 15 fields declared, then
storing data for 100 distinct customers would generate 1500 fields in the
index.
I really don't know how well would that scale.
The simplest solution is one core per customer but the same issue (memory
consumption) will rise at some point, I guess.
There must be a clever way to do that...
--
Tanguy
2012/9/26 韦震宇 <we...@win-trust.com>
> Hi, Tanguy
> I would do as your suggestion.
> Best Regards!
> Monton
> ----- Original Message -----
> From: "Tanguy Moal" <ta...@gmail.com>
> To: <so...@lucene.apache.org>; <te...@statsbiblioteket.dk>
> Sent: Tuesday, September 25, 2012 11:05 PM
> Subject: Re: How can I create about 100000 independent indexes in Solr?
>
>
> That is an interesting issue...
> I was wondering if relying on dynamic fields could be an option...
>
> Something like :
>
> * <field_name>: <field_type>
> * customer : string
> * *_field_a1 : type_a
> * *_field_a2 : type_a
> * *_field_b1 : type_b
> * ...
>
> And the prefix each field by the customer name, so for customer1, indexed
> documents are as follow :
> * customer : customer1
> * customer1_field_a1 : value for field_a1
> * customer1_field_a2 : value for field_a2
> * customer1_field_b1 : value for field_b1
> * ...
> And for customer2 :
> * customer : customer2
> * customer2_field_a1 : value for field_a1
> * customer2_field_a2 : value for field_a2
> * customer2_field_b1 : value for field_b1
> * ...
>
> This solution is simple, and helps isolating each customers fields so
> features like suggester, spellcheck, ..., things relying on frequencies
> would work (as if in a single core)
>
> I just don't how well does solr scale if the number of fields increases...
>
> Then scaling could be achieved depending on number of doc / customer and
> number of customer / core (if amount of fields consumes resources)
>
> Could that help ?
>
> --
> Tanguy
>
> 2012/9/25 Toke Eskildsen <te...@statsbiblioteket.dk>
>
> > On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote:
> > > The company I'm working in have a website to server more than 100000
> > > customers, and every customer should have it's own search cataegory.
> > > So I should create independent index for every customer.
> >
> > How many of the customers are active at any given time and how large are
> > the indexes? Depending on usage you might be able to have a limited
> > number of indexes open at any given time and opening new indexes on
> > demand.
> >
> >
>
Re: How can I create about 100000 independent indexes in Solr?
Posted by Tanguy Moal <ta...@gmail.com>.
Hello Monton,
I wanted to make sure that you understood me well : I really don't how well
does solr scale if the number of fields increases...
What I mean here is that the more distinct fields you index, the more
memory you will need.
So if in your schema, you have something like 15 fields declared, then
storing data for 100 distinct customers would generate 1500 fields in the
index.
I really don't know how well would that scale.
The simplest solution is one core per customer but the same issue (memory
consumption) will rise at some point, I guess.
There must be a clever way to do that...
--
Tanguy
2012/9/26 韦震宇 <we...@win-trust.com>
> Hi, Tanguy
> I would do as your suggestion.
> Best Regards!
> Monton
> ----- Original Message -----
> From: "Tanguy Moal" <ta...@gmail.com>
> To: <so...@lucene.apache.org>; <te...@statsbiblioteket.dk>
> Sent: Tuesday, September 25, 2012 11:05 PM
> Subject: Re: How can I create about 100000 independent indexes in Solr?
>
>
> That is an interesting issue...
> I was wondering if relying on dynamic fields could be an option...
>
> Something like :
>
> * <field_name>: <field_type>
> * customer : string
> * *_field_a1 : type_a
> * *_field_a2 : type_a
> * *_field_b1 : type_b
> * ...
>
> And the prefix each field by the customer name, so for customer1, indexed
> documents are as follow :
> * customer : customer1
> * customer1_field_a1 : value for field_a1
> * customer1_field_a2 : value for field_a2
> * customer1_field_b1 : value for field_b1
> * ...
> And for customer2 :
> * customer : customer2
> * customer2_field_a1 : value for field_a1
> * customer2_field_a2 : value for field_a2
> * customer2_field_b1 : value for field_b1
> * ...
>
> This solution is simple, and helps isolating each customers fields so
> features like suggester, spellcheck, ..., things relying on frequencies
> would work (as if in a single core)
>
> I just don't how well does solr scale if the number of fields increases...
>
> Then scaling could be achieved depending on number of doc / customer and
> number of customer / core (if amount of fields consumes resources)
>
> Could that help ?
>
> --
> Tanguy
>
> 2012/9/25 Toke Eskildsen <te...@statsbiblioteket.dk>
>
> > On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote:
> > > The company I'm working in have a website to server more than 100000
> > > customers, and every customer should have it's own search cataegory.
> > > So I should create independent index for every customer.
> >
> > How many of the customers are active at any given time and how large are
> > the indexes? Depending on usage you might be able to have a limited
> > number of indexes open at any given time and opening new indexes on
> > demand.
> >
> >
>
Re: How can I create about 100000 independent indexes in Solr?
Posted by 韦震宇 <we...@win-trust.com>.
Hi, Tanguy
I would do as your suggestion.
Best Regards!
Monton
----- Original Message -----
From: "Tanguy Moal" <ta...@gmail.com>
To: <so...@lucene.apache.org>; <te...@statsbiblioteket.dk>
Sent: Tuesday, September 25, 2012 11:05 PM
Subject: Re: How can I create about 100000 independent indexes in Solr?
That is an interesting issue...
I was wondering if relying on dynamic fields could be an option...
Something like :
* <field_name>: <field_type>
* customer : string
* *_field_a1 : type_a
* *_field_a2 : type_a
* *_field_b1 : type_b
* ...
And the prefix each field by the customer name, so for customer1, indexed
documents are as follow :
* customer : customer1
* customer1_field_a1 : value for field_a1
* customer1_field_a2 : value for field_a2
* customer1_field_b1 : value for field_b1
* ...
And for customer2 :
* customer : customer2
* customer2_field_a1 : value for field_a1
* customer2_field_a2 : value for field_a2
* customer2_field_b1 : value for field_b1
* ...
This solution is simple, and helps isolating each customers fields so
features like suggester, spellcheck, ..., things relying on frequencies
would work (as if in a single core)
I just don't how well does solr scale if the number of fields increases...
Then scaling could be achieved depending on number of doc / customer and
number of customer / core (if amount of fields consumes resources)
Could that help ?
--
Tanguy
2012/9/25 Toke Eskildsen <te...@statsbiblioteket.dk>
> On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote:
> > The company I'm working in have a website to server more than 100000
> > customers, and every customer should have it's own search cataegory.
> > So I should create independent index for every customer.
>
> How many of the customers are active at any given time and how large are
> the indexes? Depending on usage you might be able to have a limited
> number of indexes open at any given time and opening new indexes on
> demand.
>
>
Re: How can I create about 100000 independent indexes in Solr?
Posted by Tanguy Moal <ta...@gmail.com>.
That is an interesting issue...
I was wondering if relying on dynamic fields could be an option...
Something like :
* <field_name>: <field_type>
* customer : string
* *_field_a1 : type_a
* *_field_a2 : type_a
* *_field_b1 : type_b
* ...
And the prefix each field by the customer name, so for customer1, indexed
documents are as follow :
* customer : customer1
* customer1_field_a1 : value for field_a1
* customer1_field_a2 : value for field_a2
* customer1_field_b1 : value for field_b1
* ...
And for customer2 :
* customer : customer2
* customer2_field_a1 : value for field_a1
* customer2_field_a2 : value for field_a2
* customer2_field_b1 : value for field_b1
* ...
This solution is simple, and helps isolating each customers fields so
features like suggester, spellcheck, ..., things relying on frequencies
would work (as if in a single core)
I just don't how well does solr scale if the number of fields increases...
Then scaling could be achieved depending on number of doc / customer and
number of customer / core (if amount of fields consumes resources)
Could that help ?
--
Tanguy
2012/9/25 Toke Eskildsen <te...@statsbiblioteket.dk>
> On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote:
> > The company I'm working in have a website to server more than 100000
> > customers, and every customer should have it's own search cataegory.
> > So I should create independent index for every customer.
>
> How many of the customers are active at any given time and how large are
> the indexes? Depending on usage you might be able to have a limited
> number of indexes open at any given time and opening new indexes on
> demand.
>
>
Re: How can I create about 100000 independent indexes in Solr?
Posted by 韦震宇 <we...@win-trust.com>.
Hi, Eskildsen
The customer is company, so the active user is tenfold more than the customers.
I'm choicing the "a single index" and splitting index into multiple shards.
Best Regards!
Monton
----- Original Message -----
From: "Toke Eskildsen" <te...@statsbiblioteket.dk>
To: <so...@lucene.apache.org>
Sent: Tuesday, September 25, 2012 10:37 PM
Subject: Re: How can I create about 100000 independent indexes in Solr?
> On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote:
>> The company I'm working in have a website to server more than 100000
>> customers, and every customer should have it's own search cataegory.
>> So I should create independent index for every customer.
>
> How many of the customers are active at any given time and how large are
> the indexes? Depending on usage you might be able to have a limited
> number of indexes open at any given time and opening new indexes on
> demand.
>
Re: How can I create about 100000 independent indexes in Solr?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote:
> The company I'm working in have a website to server more than 100000
> customers, and every customer should have it's own search cataegory.
> So I should create independent index for every customer.
How many of the customers are active at any given time and how large are
the indexes? Depending on usage you might be able to have a limited
number of indexes open at any given time and opening new indexes on
demand.
Re: How can I create about 100000 independent indexes in Solr?
Posted by 韦震宇 <we...@win-trust.com>.
Hi, Bordis
Thank you very much for your precious experience!
I would choice the "single index" solution.
I'm sorry my name is not pronounced good in english ,you would call me monton.
Best Regards!
Monton
----- Original Message -----
From: "Christian Bordis" <c....@epages.com>
To: <so...@lucene.apache.org>
Sent: Tuesday, September 25, 2012 9:18 PM
Subject: AW: How can I create about 100000 independent indexes in Solr?
> Hi weizhenyu! (can you write down your name in roman latters)
>
> We also tried to provide dedicated cores for each customer in one of our early system design to get maximum config possibilities.
> 40k Webshops => 40k Solr Cores, which didn't work out.
>
> Problems we encountered:
> - Threads: Solr create one thread to interact with each core in 3.x of solr , 40k therads will kill any server & performance
> - FileSystem: Each Core needed number of files 30 -100 files depending on settings etc. 40.000 x 100 file handles was not an option for the CentOS (and most other OS)
> - Managing 100k configs is problem of its own
>
> You may look for other solution e.g. split the user base in small number of Cores by use case and try to cover their needs. (btw the solrconfig.xml, schema.xml ultra flexible most likely the cover 95% of your requirements)
>
> Regards,
> Christian Bordis
>
> -----Ursprüngliche Nachricht-----
> Von: 韦震宇 [mailto:weizhenyu@win-trust.com]
> Gesendet: Dienstag, 25. September 2012 04:21
> An: solr-user@lucene.apache.org
> Betreff: How can I create about 100000 independent indexes in Solr?
>
> Dear all,
> The company I'm working in have a website to server more than 100000 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer.
> The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes.
> I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution.
> The other solution "Flattening data into a single index" is a choice, but i think it's best to keep all indexes indepent.
> Could you tell me how to create about 100000 independent indexes in Solr?
> Thank you all for reply!
>
>
AW: How can I create about 100000 independent indexes in Solr?
Posted by Christian Bordis <c....@epages.com>.
Hi weizhenyu! (can you write down your name in roman latters)
We also tried to provide dedicated cores for each customer in one of our early system design to get maximum config possibilities.
40k Webshops => 40k Solr Cores, which didn't work out.
Problems we encountered:
- Threads: Solr create one thread to interact with each core in 3.x of solr , 40k therads will kill any server & performance
- FileSystem: Each Core needed number of files 30 -100 files depending on settings etc. 40.000 x 100 file handles was not an option for the CentOS (and most other OS)
- Managing 100k configs is problem of its own
You may look for other solution e.g. split the user base in small number of Cores by use case and try to cover their needs. (btw the solrconfig.xml, schema.xml ultra flexible most likely the cover 95% of your requirements)
Regards,
Christian Bordis
-----Ursprüngliche Nachricht-----
Von: 韦震宇 [mailto:weizhenyu@win-trust.com]
Gesendet: Dienstag, 25. September 2012 04:21
An: solr-user@lucene.apache.org
Betreff: How can I create about 100000 independent indexes in Solr?
Dear all,
The company I'm working in have a website to server more than 100000 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer.
The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes.
I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution.
The other solution "Flattening data into a single index" is a choice, but i think it's best to keep all indexes indepent.
Could you tell me how to create about 100000 independent indexes in Solr?
Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Posted by 韦震宇 <we...@win-trust.com>.
Hi,Otis
Thank you for your suggestion! I'm considering the "a single index" solution.
Best Regards!
Monton
----- Original Message -----
From: "Otis Gospodnetic" <ot...@gmail.com>
To: <so...@lucene.apache.org>
Sent: Wednesday, September 26, 2012 10:26 AM
Subject: Re: How can I create about 100000 independent indexes in Solr?
Hi,
Have a look at ElasticSearch. While it's probably no better at
handling 100K cores than Solr it, ES has document and query routing
functionality that can help you route documents and queries to
specific shards, thus avoiding needing to query all 100K users' data.
I think Solr doesn't have this feature yet, but I'm sure it will in
the future.
Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html
On Tue, Sep 25, 2012 at 10:20 PM, 韦震宇 <we...@win-trust.com> wrote:
> Hi,Tom
> I'm prefer performance, so if the performance of "single index" solution is better than "mutilcore" solution, I would choice the former.
> Best Regards!
> ----- Original Message -----
> From: "Tom Mortimer" <to...@gmail.com>
> To: <so...@lucene.apache.org>
> Sent: Tuesday, September 25, 2012 6:06 PM
> Subject: Re: How can I create about 100000 independent indexes in Solr?
>
>
> Hi,
>
> Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries?
>
> Tom
>
> On 25 Sep 2012, at 03:21, 韦震宇 <we...@win-trust.com> wrote:
>
>> Dear all,
>> The company I'm working in have a website to server more than 100000 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer.
>> The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes.
>> I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution.
>> The other solution "Flattening data into a single index" is a choice, but i think it's best to keep all indexes indepent.
>> Could you tell me how to create about 100000 independent indexes in Solr?
>> Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Have a look at ElasticSearch. While it's probably no better at
handling 100K cores than Solr it, ES has document and query routing
functionality that can help you route documents and queries to
specific shards, thus avoiding needing to query all 100K users' data.
I think Solr doesn't have this feature yet, but I'm sure it will in
the future.
Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html
On Tue, Sep 25, 2012 at 10:20 PM, 韦震宇 <we...@win-trust.com> wrote:
> Hi,Tom
> I'm prefer performance, so if the performance of "single index" solution is better than "mutilcore" solution, I would choice the former.
> Best Regards!
> ----- Original Message -----
> From: "Tom Mortimer" <to...@gmail.com>
> To: <so...@lucene.apache.org>
> Sent: Tuesday, September 25, 2012 6:06 PM
> Subject: Re: How can I create about 100000 independent indexes in Solr?
>
>
> Hi,
>
> Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries?
>
> Tom
>
> On 25 Sep 2012, at 03:21, 韦震宇 <we...@win-trust.com> wrote:
>
>> Dear all,
>> The company I'm working in have a website to server more than 100000 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer.
>> The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes.
>> I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution.
>> The other solution "Flattening data into a single index" is a choice, but i think it's best to keep all indexes indepent.
>> Could you tell me how to create about 100000 independent indexes in Solr?
>> Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Posted by 韦震宇 <we...@win-trust.com>.
Hi,Tom
I'm prefer performance, so if the performance of "single index" solution is better than "mutilcore" solution, I would choice the former.
Best Regards!
----- Original Message -----
From: "Tom Mortimer" <to...@gmail.com>
To: <so...@lucene.apache.org>
Sent: Tuesday, September 25, 2012 6:06 PM
Subject: Re: How can I create about 100000 independent indexes in Solr?
Hi,
Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries?
Tom
On 25 Sep 2012, at 03:21, 韦震宇 <we...@win-trust.com> wrote:
> Dear all,
> The company I'm working in have a website to server more than 100000 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer.
> The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes.
> I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution.
> The other solution "Flattening data into a single index" is a choice, but i think it's best to keep all indexes indepent.
> Could you tell me how to create about 100000 independent indexes in Solr?
> Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Tue, 2012-09-25 at 12:06 +0200, Tom Mortimer wrote:
> Why do you think that the indexes should be independent? What would be
> the problem with using a single index and filter queries?
Ranking will be done with regard to the full corpus instead of just the
single customer. Whether that is an issue or not depends on the content.
e.g. for email archives, the single index will not work very well.
- Toke Eskildsen, State and University Library, Denmark
Re: How can I create about 100000 independent indexes in Solr?
Posted by Tom Mortimer <to...@gmail.com>.
Hi,
Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries?
Tom
On 25 Sep 2012, at 03:21, 韦震宇 <we...@win-trust.com> wrote:
> Dear all,
> The company I'm working in have a website to server more than 100000 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer.
> The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes.
> I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution.
> The other solution "Flattening data into a single index" is a choice, but i think it's best to keep all indexes indepent.
> Could you tell me how to create about 100000 independent indexes in Solr?
> Thank you all for reply!