You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "kowish.adamosh" <ko...@gmail.com> on 2013/07/02 16:35:11 UTC
Solr cloud date based paritioning
Hi guys!
I have simple use case to implement but I have problem with date based
partitioning... Here are some rules:
1. At the beginning I have to create huge index (10GB) based on one db
table.
2. Every day I have to update this index.
3. 99,999% are queries based on date field (*data from last 2 months*).
So my idea was to create partitions by month an provide month-based
partition in query like in example in documentation:
http://localhost:8983/solr/collection1/select?shards=shard_200812,shard_200912,shard_201001
I would provide shards only from 2 months to gain nice performance.
Questions are: how can I create month-based partitions? Is it possible to
create new shard on each new month are update delta data only to this shard?
Examples are very welcome. I read documentation a few times and can't find
answers...
Thanks!
Kowish
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-cloud-date-based-paritioning-tp4074729.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr cloud date based paritioning
Posted by Gora Mohanty <go...@mimirtech.com>.
On 2 July 2013 20:05, kowish.adamosh <ko...@gmail.com> wrote:
> Hi guys!
>
> I have simple use case to implement but I have problem with date based
> partitioning... Here are some rules:
>
> 1. At the beginning I have to create huge index (10GB) based on one db
> table.
> 2. Every day I have to update this index.
> 3. 99,999% are queries based on date field (*data from last 2 months*).
[...]
Before you start complicating things, have you measured the
performance of having everything in one shard? It is quite
likely that a 10GB index would have adequate performance
on reasonable hardware. Your mileage may vary, but I would
try to measure the performance from a single index first.
Regards,
Gora
Re: Solr cloud date based paritioning
Posted by Jan Høydahl <ja...@cominvent.com>.
Not yet, you may look at https://issues.apache.org/jira/browse/SOLR-4221 for possible future solution.
For the time being you could use Implicit routing, i.e. your application sends the docs directly to the correct shard, so every month your app code would decide to switch indexing to a new node, and the same with searching. You achieve implicit routing by creating the collection without a numShards parameter.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
2. juli 2013 kl. 22:11 skrev kowish.adamosh <ko...@gmail.com>:
> Sure, I'ill measure results and come back if results will be unsatisfactory.
> Thanks very much for advice.
>
> Out of curiosity: is there any way to partition shards (logical and
> physical) by specified value of specified field?
>
> Kowish
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-cloud-date-based-paritioning-tp4074729p4074899.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr cloud date based paritioning
Posted by Mark Miller <ma...@gmail.com>.
The strategy doesn't require putting all the recent data on a single node.
What has been suggested is collection based - the most recent data will simply be in it's own collection, that may or may not be on a single node.
This is pretty much always going to be advantageous for time series data. You can manage things as Otis said - even having a collection for the past hour if that's what you need to do. That last hour can be distributed over as many nodes as you need.
Older collections will be stable and can be heavily cached, perhaps serve queries more slowly, and can be easily merged together or removed from the system.
Nothing about it is automatically good, but this is the type of architecture you pretty much always want with time series data if you really want to scale. There are lot's of little twists around it and different tradeoffs you can make, but by and large, the main concepts are the same.
There are many things to consider when building this architecture - for example, you may want to carefully control collection placement rather than accepting the std random distribution across nodes.
Eventually it should become fairly simple to implement this pattern with a single collection as the custom sharding features advance.
- Mark
On Jul 6, 2013, at 7:23 AM, Erick Erickson <er...@gmail.com> wrote:
> Not saying it's always one way or the other, just
> that one shouldn't automatically _assume_
> putting the most recent data on a single node
> is automatically good. It may well be, but
> not in all cases.
>
>
>
>
> On Wed, Jul 3, 2013 at 12:21 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> Exactly. And the newest shard can also be kept small (e.g. maybe just
>> last 12h is OK to hit first and dig deeper only if you can't find
>> enough stories in the last 12h), which means it will fit in memory and
>> be crazy fast.
>>
>> Otis
>> --
>> Solr & ElasticSearch Support -- http://sematext.com/
>> Performance Monitoring -- http://sematext.com/spm
>>
>>
>>
>> On Wed, Jul 3, 2013 at 10:11 AM, Mark Miller <ma...@gmail.com>
>> wrote:
>>>
>>> On Jul 3, 2013, at 7:47 AM, Erick Erickson <er...@gmail.com>
>> wrote:
>>>
>>>> Usually most people
>>>> care about today's news, and a hot story will
>>>> generate lots of queries, all of which are serviced
>>>> by today's shard.
>>>
>>> That's really the whole point though - rather than slamming your whole
>> cluster with every search, the majority of people are just searching today
>> - which will have only a fraction of the data and will be able to hold up
>> very well to a large load. This is also how you can do really fast NRT on a
>> huge data set - it only has to happen on todays shard.
>>>
>>> News sites have been using the trick forever with Lucene.
>>>
>>> - Mark
>>
Re: Solr cloud date based paritioning
Posted by Erick Erickson <er...@gmail.com>.
Not saying it's always one way or the other, just
that one shouldn't automatically _assume_
putting the most recent data on a single node
is automatically good. It may well be, but
not in all cases.
On Wed, Jul 3, 2013 at 12:21 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Exactly. And the newest shard can also be kept small (e.g. maybe just
> last 12h is OK to hit first and dig deeper only if you can't find
> enough stories in the last 12h), which means it will fit in memory and
> be crazy fast.
>
> Otis
> --
> Solr & ElasticSearch Support -- http://sematext.com/
> Performance Monitoring -- http://sematext.com/spm
>
>
>
> On Wed, Jul 3, 2013 at 10:11 AM, Mark Miller <ma...@gmail.com>
> wrote:
> >
> > On Jul 3, 2013, at 7:47 AM, Erick Erickson <er...@gmail.com>
> wrote:
> >
> >> Usually most people
> >> care about today's news, and a hot story will
> >> generate lots of queries, all of which are serviced
> >> by today's shard.
> >
> > That's really the whole point though - rather than slamming your whole
> cluster with every search, the majority of people are just searching today
> - which will have only a fraction of the data and will be able to hold up
> very well to a large load. This is also how you can do really fast NRT on a
> huge data set - it only has to happen on todays shard.
> >
> > News sites have been using the trick forever with Lucene.
> >
> > - Mark
>
Re: Solr cloud date based paritioning
Posted by Otis Gospodnetic <ot...@gmail.com>.
Exactly. And the newest shard can also be kept small (e.g. maybe just
last 12h is OK to hit first and dig deeper only if you can't find
enough stories in the last 12h), which means it will fit in memory and
be crazy fast.
Otis
--
Solr & ElasticSearch Support -- http://sematext.com/
Performance Monitoring -- http://sematext.com/spm
On Wed, Jul 3, 2013 at 10:11 AM, Mark Miller <ma...@gmail.com> wrote:
>
> On Jul 3, 2013, at 7:47 AM, Erick Erickson <er...@gmail.com> wrote:
>
>> Usually most people
>> care about today's news, and a hot story will
>> generate lots of queries, all of which are serviced
>> by today's shard.
>
> That's really the whole point though - rather than slamming your whole cluster with every search, the majority of people are just searching today - which will have only a fraction of the data and will be able to hold up very well to a large load. This is also how you can do really fast NRT on a huge data set - it only has to happen on todays shard.
>
> News sites have been using the trick forever with Lucene.
>
> - Mark
Re: Solr cloud date based paritioning
Posted by Mark Miller <ma...@gmail.com>.
On Jul 3, 2013, at 7:47 AM, Erick Erickson <er...@gmail.com> wrote:
> Usually most people
> care about today's news, and a hot story will
> generate lots of queries, all of which are serviced
> by today's shard.
That's really the whole point though - rather than slamming your whole cluster with every search, the majority of people are just searching today - which will have only a fraction of the data and will be able to hold up very well to a large load. This is also how you can do really fast NRT on a huge data set - it only has to happen on todays shard.
News sites have been using the trick forever with Lucene.
- Mark
Re: Solr cloud date based paritioning
Posted by Erick Erickson <er...@gmail.com>.
Not automatically as far as I know. You can use
custom routing to achieve this though.
That said, be careful before you jump this way.
While it _seems_ elegant, something like have
one shard per day for the retention period (say 30 days)
and just rotate the oldest one out, it may not be
best. Consider the news case. Usually most people
care about today's news, and a hot story will
generate lots of queries, all of which are serviced
by today's shard. Spreading your queries
out amongst all 30 shards may actually be a better
user experience, then to purge old data just
do a delete-by-query on a datestamp, something
like timestamp:[* TO NOW-30DAYS].
Don't know if it applies to your particular use case
or not, but....
Erick
On Wed, Jul 3, 2013 at 2:53 AM, kowish.adamosh <ko...@gmail.com>wrote:
> Otis,
>
> Is it implemented? I can find only unresolved JIRA issues.
>
> Kowish
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-cloud-date-based-paritioning-tp4074729p4074993.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr cloud date based paritioning
Posted by "kowish.adamosh" <ko...@gmail.com>.
Otis,
Is it implemented? I can find only unresolved JIRA issues.
Kowish
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-cloud-date-based-paritioning-tp4074729p4074993.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr cloud date based paritioning
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Yes, search for keywords shard and route in ML archives and JIRA.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Jul 2, 2013 4:12 PM, "kowish.adamosh" <ko...@gmail.com> wrote:
> Sure, I'ill measure results and come back if results will be
> unsatisfactory.
> Thanks very much for advice.
>
> Out of curiosity: is there any way to partition shards (logical and
> physical) by specified value of specified field?
>
> Kowish
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-cloud-date-based-paritioning-tp4074729p4074899.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr cloud date based paritioning
Posted by "kowish.adamosh" <ko...@gmail.com>.
Sure, I'ill measure results and come back if results will be unsatisfactory.
Thanks very much for advice.
Out of curiosity: is there any way to partition shards (logical and
physical) by specified value of specified field?
Kowish
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-cloud-date-based-paritioning-tp4074729p4074899.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr cloud date based paritioning
Posted by Gora Mohanty <go...@mimirtech.com>.
On 2 July 2013 22:35, kowish.adamosh <ko...@gmail.com> wrote:
> Thanks!
>
> I have very limited response time (max 100ms) therefore sharding is a must.
Really? "Sharding is a must" without any measurements to
validate that assertion? I am not sure what advice to give
you if you seem determined to ignore any, but as a touch
point, in the days of Solr 1.4 (much improved perfomance
since then), out of the box we used to get an average time
of well under 100ms for queries with > 50 simultaneous
users on an index with *everything* stored, and an index
size of > 80 GB. This was admittedly non-scientific, as the
cache enters significantly into the equation, but I will urge
you again: Try measuring things before adding bells and
whistles.
Regards,
Gora
Re: Solr cloud date based paritioning
Posted by "kowish.adamosh" <ko...@gmail.com>.
Thanks!
I have very limited response time (max 100ms) therefore sharding is a must.
Data also have trend to grow up to tens of gigs.
Is there any way how to create new logical shard in runtime? I want to
logically partition my data by date. I'm still wondering how is implemented
example from documentation:
/
Query specific shard ids of the (implicit) collection. In this example, the
user has partitioned the index by date, creating a new shard every month:
http://localhost:8983/solr/collection1/select?shards=shard_200812,shard_200912,shard_201001/
Even in first full load I don't know how to do it... In all examples I can
see that data are distributed physically by uniqeId % coreNum. Are there
some examples of custom (i.e. date based) sharding strategy?
I can see that in JIRA: https://issues.apache.org/jira/browse/SOLR-2592
there is something that may help but I can't find anything in documentation.
Thanks for help!
Kowish
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-cloud-date-based-paritioning-tp4074729p4074823.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr cloud date based paritioning
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
There is nothing automatic that I know of that will create shards (or
maybe you mean SolrCloud Collections?) every month. You can do that
in your application, though, just create the Collection via the API.
You can make use of aliases to have something like "last2months" alias
point to your last 2 Collections. You would "shift" this alias every
month after you create your new Collection. Of course, right after
the shift, you would really be searching only 1 month's worth of data,
so you may want to allow searching across last 3 Collections instead,
optionally enforcing/limiting query to last 2 months based on document
date and a range query.
Otis
--
Solr & ElasticSearch Support -- http://sematext.com/
Performance Monitoring -- http://sematext.com/spm
On Tue, Jul 2, 2013 at 10:35 AM, kowish.adamosh
<ko...@gmail.com> wrote:
> Hi guys!
>
> I have simple use case to implement but I have problem with date based
> partitioning... Here are some rules:
>
> 1. At the beginning I have to create huge index (10GB) based on one db
> table.
> 2. Every day I have to update this index.
> 3. 99,999% are queries based on date field (*data from last 2 months*).
>
> So my idea was to create partitions by month an provide month-based
> partition in query like in example in documentation:
> http://localhost:8983/solr/collection1/select?shards=shard_200812,shard_200912,shard_201001
>
> I would provide shards only from 2 months to gain nice performance.
>
> Questions are: how can I create month-based partitions? Is it possible to
> create new shard on each new month are update delta data only to this shard?
>
> Examples are very welcome. I read documentation a few times and can't find
> answers...
>
> Thanks!
> Kowish
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-cloud-date-based-paritioning-tp4074729.html
> Sent from the Solr - User mailing list archive at Nabble.com.