You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Long Yan <lo...@robotron.de> on 2015/09/02 11:17:39 UTC
Strange behavior of solr
Hey,
I have created a core with
bin\solr create -c mycore
I want to index the csv sample files from solr-5.2.1
If I index film.csv under solr-5.2.1\example\films\, solr can only index this file until the line
"2046,Wong Kar-wai,Romance Film|Fantasy|Science Fiction|Drama,,/en/2046_2004,2004-05-20"
But if I at first index books.csv under solr-5.2.1\example\exampledocs and then index film.csv, solr can index all lines in film.csv
Why?
Regards
Long Yan
Re: concept and choice: custom sharding or auto sharding?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 9/2/2015 9:19 AM, scott chu wrote:
> Mail
> Do you mean I only have to put 10M documents in one index and copy
> it to many slaves in a classic Solr master-slave architecture to
> provide querying serivce on internet, and it won't have obvious
> downgrade of query performance? But I did have add 1M document into
> one index on master and provide 2 slaves to serve querying service on
> internet, the query performance is kinda sad. Why do you say: "at 10M
> documents there's rarely a need to shard at all?" Do I provide too few
> slaves? What amount of documents is suitable for a need for shard in
> SolrCloud?
Lucene has exactly one hard and unbreakable limit, and it is the number
of documents you can have in a single index (core/shard for Solr). That
limit is just over 2.1 billion documents. The actual limiting factor is
the maximum value of an integer in Java. Because deleted documents are
counted when this limit is considered, you shouldn't go over 1 billion
active documents per shard, but the *practical* recommendation for shard
size is much lower than that.
For various reasons, some of which are very technical and boring, the
general advice is to not exceed about 100 million documents per shard.
Some setups can handle more docs per shard, some require a lot less.
There are no quick answers or hard rules. You may have been given this
URL before:
https://lucidworks.com/blog/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
There are sometimes reasons to shard a very small index. This is the
correct path when the index is not very busy and you want to take
advantage of a surplus of CPU power to make each query faster. In this
situation, you will probably end up with multiple shards per server, so
when a query comes in, multiple CPUs on the machine can handle the shard
queries in parallel. If the index is handling a lot of requests per
second, then you want that CPU power used for handling the load, not
speeding up a single query. For high-load situations, one shard per
physical server is desirable.
Thanks,
Shawn
Re: Re: Re: Re: concept and choice: custom sharding or auto sharding?
Posted by scott chu <sc...@udngroup.com>.
solr-user，妳好
Sorry ,wrong again. Auto sharding is not implicit router.
----- Original Message -----
From: scott chu
To: solr-user
Date: 2015-09-02, 23:50:20
Subject: Re: Re: Re: concept and choice: custom sharding or auto sharding?
solr-user，妳好
Thanks! I'll go back to check my old environment and that article is really helpful.
BTW, I think I got wrong about compositeID. In the reference guide, it said compositeID needs numShards. That means what I describe in question 5 seems wrong cause I intend to plan one shard one whole year news article and I thought SolrCloud will create new shard for me itself when I add new year's articles. But since compositeID needs to specify numShards first, there's no way I can know how many years I will put in SolrCloud in advance . IT looks like if I want to use SolrCloud afte all, I may have to use auto sharding (i.e. implicit router).
----- Original Message -----
From: Erick Erickson
To: solr-user
Date: 2015-09-02, 23:30:53
Subject: Re: Re: concept and choice: custom sharding or auto sharding?
bq: Why do you say: "at 10M documents there's rarely a need to shard at all?"
Because I routinely see 50M docs on a single node and I've seen over 300M docs
on a single node with sub-second responses. So if you're saying that
you see poor
performance at 1M docs then I suspect there's something radically
wrong with your
setup. Too little memory, very bad query patterns, whatever. If my
suspicion is true,
then sharding will just mask the underlying problem.
You need to quantify your performance concerns. It's one thing to say
"my node satisfies 50 queries-per-second with 500ms response time" and
another to say "My queries take 5,000 ms".
In the first case, you do indeed need to add more servers to increase QPS if
you need 500 QPS. And adding more slaves is the best way to do that.
In the second, you need to understand the slowdown because sharding
will be a band-aid.
This might help:
https://wiki.apache.org/solr/SolrPerformanceProblems
Best,
Erick
On Wed, Sep 2, 2015 at 8:19 AM, scott chu <sc...@udngroup.com> wrote:
>
> solr-user，妳好
>
> Do you mean I only have to put 10M documents in one index and copy it to
> many slaves in a classic Solr master-slave architecture to provide querying
> serivce on internet, and it won't have obvious downgrade of query
> performance? But I did have add 1M document into one index on master and
> provide 2 slaves to serve querying service on internet, the query
> performance is kinda sad. Why do you say: "at 10M documents there's rarely a
> need to shard at all?" Do I provide too few slaves? What amount of documents
> is suitable for a need for shard in SolrCloud?
>
> ----- Original Message -----
>
> From: Erick Erickson
> To: solr-user
> Date: 2015-09-02, 23:00:29
> Subject: Re: concept and choice: custom sharding or auto sharding?
>
> Frankly, at 10M documents there's rarely a need to shard at all.
> Why do you think you need to? This seems like adding
> complexity for no good reason. Sharding should only really
> be used when you have too many documents to fit on a single
> shard as it adds some overhead, restricts some
> possibilities (cross-core join for instance, a couple of
> grouping options don't work in distributed mode etc.).
>
> You can still run SolrCloud and have it manage multiple
> _replicas_ of a single shard for HA/DR.
>
> So this seems like an XY problem, you're asking specific
> questions about shard routing because you think it'll
> solve some problem without telling us what the problem
> is.
>
> Best,
> Erick
>
> On Wed, Sep 2, 2015 at 7:47 AM, scott chu <sc...@udngroup.com> wrote:
>> I post a question on Stackoverflow
>> http://stackoverflow.com/questions/32343813/custom-sharding-or-auto-sharding-on-solrcloud:
>> However, since this is a mail-list, I repost the question below to request
>> for suggestion and more subtle concept of SolrCloud's behavior on document
>> routing.
>> I want to establish a SolrCloud clsuter for over 10 millions of news
>> articles. After reading this article in Apache Solr Refernce guide: Shards
>> and Indexing Data in SolrCloud, I have a plan as follows:
>> Add prefix ED2001! to document ID where ED means some newspaper source and
>> 2001 is the year part in published date of news article, i.e. I want to put
>> all news articles of specific news paper source published in specific year
>> to a shard.
>> Create collection with router.name set to compositeID.
>> Add documents?
>> Query Collection?
>> Practically, I got some questions:
>> How to add doucments based on this plan? Do I have to specify special
>> parameters when updating the collection/core?
>> Is this called "custom sharding"? If not, what is "custom sharding"?
>> Is auto sharding a better choice for my case since there's a
>> shard-splitting feature for auto sharding when the shard is too big?
>> Can I query without _router_ parameter?
>> EDIT @ 2015/9/2:
>> This is how I think SolrCloud will do: "The amount of news articles of
>> specific newspaper source of specific year tends to be around a fix number,
>> e.g. Every year ED has around 80,000 articles, so each shard's size won't
>> increase dramatically. For the next year's news articles of ED, I only have
>> to add prefix 'ED2016!' to document ID, SolrCloud will create a new shard
>> for me (which contains all ED2016 articles), and later the Leader will
>> spread the replica of this new shard to other nodes (per replica per node
>> other than leader?)". Am I right? If yes, it seems no need for
>> shard-splitting.
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6086 / 病毒庫: 4409/10562 - 發佈日期: 09/02/15
>
>
>
>
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6086 / 病毒庫: 4409/10562 - 發佈日期: 09/02/15
Re: Re: Re: concept and choice: custom sharding or auto sharding?
Posted by scott chu <sc...@udngroup.com>.
solr-user，妳好
Thanks! I'll go back to check my old environment and that article is really helpful.
BTW, I think I got wrong about compositeID. In the reference guide, it said compositeID needs numShards. That means what I describe in question 5 seems wrong cause I intend to plan one shard one whole year news article and I thought SolrCloud will create new shard for me itself when I add new year's articles. But since compositeID needs to specify numShards first, there's no way I can know how many years I will put in SolrCloud in advance . IT looks like if I want to use SolrCloud afte all, I may have to use auto sharding (i.e. implicit router).
----- Original Message -----
From: Erick Erickson
To: solr-user
Date: 2015-09-02, 23:30:53
Subject: Re: Re: concept and choice: custom sharding or auto sharding?
bq: Why do you say: "at 10M documents there's rarely a need to shard at all?"
Because I routinely see 50M docs on a single node and I've seen over 300M docs
on a single node with sub-second responses. So if you're saying that
you see poor
performance at 1M docs then I suspect there's something radically
wrong with your
setup. Too little memory, very bad query patterns, whatever. If my
suspicion is true,
then sharding will just mask the underlying problem.
You need to quantify your performance concerns. It's one thing to say
"my node satisfies 50 queries-per-second with 500ms response time" and
another to say "My queries take 5,000 ms".
In the first case, you do indeed need to add more servers to increase QPS if
you need 500 QPS. And adding more slaves is the best way to do that.
In the second, you need to understand the slowdown because sharding
will be a band-aid.
This might help:
https://wiki.apache.org/solr/SolrPerformanceProblems
Best,
Erick
On Wed, Sep 2, 2015 at 8:19 AM, scott chu <sc...@udngroup.com> wrote:
>
> solr-user，妳好
>
> Do you mean I only have to put 10M documents in one index and copy it to
> many slaves in a classic Solr master-slave architecture to provide querying
> serivce on internet, and it won't have obvious downgrade of query
> performance? But I did have add 1M document into one index on master and
> provide 2 slaves to serve querying service on internet, the query
> performance is kinda sad. Why do you say: "at 10M documents there's rarely a
> need to shard at all?" Do I provide too few slaves? What amount of documents
> is suitable for a need for shard in SolrCloud?
>
> ----- Original Message -----
>
> From: Erick Erickson
> To: solr-user
> Date: 2015-09-02, 23:00:29
> Subject: Re: concept and choice: custom sharding or auto sharding?
>
> Frankly, at 10M documents there's rarely a need to shard at all.
> Why do you think you need to? This seems like adding
> complexity for no good reason. Sharding should only really
> be used when you have too many documents to fit on a single
> shard as it adds some overhead, restricts some
> possibilities (cross-core join for instance, a couple of
> grouping options don't work in distributed mode etc.).
>
> You can still run SolrCloud and have it manage multiple
> _replicas_ of a single shard for HA/DR.
>
> So this seems like an XY problem, you're asking specific
> questions about shard routing because you think it'll
> solve some problem without telling us what the problem
> is.
>
> Best,
> Erick
>
> On Wed, Sep 2, 2015 at 7:47 AM, scott chu <sc...@udngroup.com> wrote:
>> I post a question on Stackoverflow
>> http://stackoverflow.com/questions/32343813/custom-sharding-or-auto-sharding-on-solrcloud:
>> However, since this is a mail-list, I repost the question below to request
>> for suggestion and more subtle concept of SolrCloud's behavior on document
>> routing.
>> I want to establish a SolrCloud clsuter for over 10 millions of news
>> articles. After reading this article in Apache Solr Refernce guide: Shards
>> and Indexing Data in SolrCloud, I have a plan as follows:
>> Add prefix ED2001! to document ID where ED means some newspaper source and
>> 2001 is the year part in published date of news article, i.e. I want to put
>> all news articles of specific news paper source published in specific year
>> to a shard.
>> Create collection with router.name set to compositeID.
>> Add documents?
>> Query Collection?
>> Practically, I got some questions:
>> How to add doucments based on this plan? Do I have to specify special
>> parameters when updating the collection/core?
>> Is this called "custom sharding"? If not, what is "custom sharding"?
>> Is auto sharding a better choice for my case since there's a
>> shard-splitting feature for auto sharding when the shard is too big?
>> Can I query without _router_ parameter?
>> EDIT @ 2015/9/2:
>> This is how I think SolrCloud will do: "The amount of news articles of
>> specific newspaper source of specific year tends to be around a fix number,
>> e.g. Every year ED has around 80,000 articles, so each shard's size won't
>> increase dramatically. For the next year's news articles of ED, I only have
>> to add prefix 'ED2016!' to document ID, SolrCloud will create a new shard
>> for me (which contains all ED2016 articles), and later the Leader will
>> spread the replica of this new shard to other nodes (per replica per node
>> other than leader?)". Am I right? If yes, it seems no need for
>> shard-splitting.
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6086 / 病毒庫: 4409/10562 - 發佈日期: 09/02/15
>
>
>
>
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6086 / 病毒庫: 4409/10562 - 發佈日期: 09/02/15
Re: concept and choice: custom sharding or auto sharding?
Posted by scott chu <sc...@udngroup.com>.
Too busy these days on work. I'd like to continue this topic talk.
I got 10M＋ tradition-chinese news articles. Due to lack of time to write my own traditional-Chinese tokenizer, I use CJK tokenizer. However, CJK uses bigram and thus will create a very large index on my case. I don't know if I should jump directly to SolrCloud, or as Erik suggest. just create master-slave architecture with better tuninb of perfomance, to buid a 10M+ tradition chinese Solrbase?
Furthermore, if anyone has any good recommendation of traditional Chinese tokenizer?
----- Original Message -----
From: Erick Erickson
To: solr-user
Date: 2015-09-04, 01:47:23
Subject: Re: Re: Re: Re: Re: concept and choice: custom sharding or auto sharding?
Ah, that may make my suggestions unworkable re: just reindexing.
Still, how much time are we talking about here? I've very often found
that indexing performance isn't gated by the Solr processing, but by
whatever is feeding Solr. A quick test is to fire up your indexing
and see if the CPU utilization by Solr is very high. As Toke said,
though, if you're using DIH you're out of luck.
Here's an article to get you started with SolrJ:
http://lucidworks.com/blog/indexing-with-solrj/
Best,
Erick
On Thu, Sep 3, 2015 at 10:26 AM, Toke Eskildsen <te...@statsbiblioteket.dk> wrote:
> scott chu <sc...@udngroup.com> wrote:
>> No, both. But first I have to face the indexing performance problem.
>> Where can I see information about concurrent/parallel indexing on Solr?
>
> Depends on how you index. If you use a Java program,
> http://lucene.apache.org/solr/5_2_0/solr-solrj/index.html?org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.html
> seems to do the trick (I haven't tried that one myself).
>
> If you are sending updates using curl or similar, you just need to start more processes doing that.
>
> If you are using DataImportHandler, I think you are out of luck. As far as I know, it does not support multiple index threads.
>
> - Toke Eskildsen
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6086 / 病毒庫: 4409/10571 - 發佈日期: 09/03/15
Re: Re: Re: Re: Re: concept and choice: custom sharding or auto sharding?
Posted by Erick Erickson <er...@gmail.com>.
Ah, that may make my suggestions unworkable re: just reindexing.
Still, how much time are we talking about here? I've very often found
that indexing performance isn't gated by the Solr processing, but by
whatever is feeding Solr. A quick test is to fire up your indexing
and see if the CPU utilization by Solr is very high. As Toke said,
though, if you're using DIH you're out of luck.
Here's an article to get you started with SolrJ:
http://lucidworks.com/blog/indexing-with-solrj/
Best,
Erick
On Thu, Sep 3, 2015 at 10:26 AM, Toke Eskildsen <te...@statsbiblioteket.dk> wrote:
> scott chu <sc...@udngroup.com> wrote:
>> No, both. But first I have to face the indexing performance problem.
>> Where can I see information about concurrent/parallel indexing on Solr?
>
> Depends on how you index. If you use a Java program,
> http://lucene.apache.org/solr/5_2_0/solr-solrj/index.html?org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.html
> seems to do the trick (I haven't tried that one myself).
>
> If you are sending updates using curl or similar, you just need to start more processes doing that.
>
> If you are using DataImportHandler, I think you are out of luck. As far as I know, it does not support multiple index threads.
>
> - Toke Eskildsen
Re: Re: Re: Re: Re: Re: concept and choice: custom sharding or autosharding?
Posted by scott chu <sc...@udngroup.com>.
solr-user，妳好
Thanks for the info. I also find this: Parallel indexing for Apache Solr Search Integration | Nick Veenhof
http://nickveenhof.be/blog/parallel-indexing-apache-solr-search-integration
----- Original Message -----
From: Toke Eskildsen
To: solr_user lucene_apache
Date: 2015-09-04, 01:26:38
Subject: Re: Re: Re: Re: Re: concept and choice: custom sharding or autosharding?
scott chu <sc...@udngroup.com> wrote:
> No, both. But first I have to face the indexing performance problem.
> Where can I see information about concurrent/parallel indexing on Solr?
Depends on how you index. If you use a Java program,
http://lucene.apache.org/solr/5_2_0/solr-solrj/index.html?org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.html
seems to do the trick (I haven't tried that one myself).
If you are sending updates using curl or similar, you just need to start more processes doing that.
If you are using DataImportHandler, I think you are out of luck. As far as I know, it does not support multiple index threads.
- Toke Eskildsen
-----
???????????
??? AVG ?? - www.avg.com
??: 2015.0.6086 / ???: 4409/10567 - ????: 09/03/15
Re: Re: Re: Re: Re: concept and choice: custom sharding or auto
sharding?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
scott chu <sc...@udngroup.com> wrote:
> No, both. But first I have to face the indexing performance problem.
> Where can I see information about concurrent/parallel indexing on Solr?
Depends on how you index. If you use a Java program,
http://lucene.apache.org/solr/5_2_0/solr-solrj/index.html?org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.html
seems to do the trick (I haven't tried that one myself).
If you are sending updates using curl or similar, you just need to start more processes doing that.
If you are using DataImportHandler, I think you are out of luck. As far as I know, it does not support multiple index threads.
- Toke Eskildsen
Re: Re: Re: Re: Re: concept and choice: custom sharding or auto sharding?
Posted by scott chu <sc...@udngroup.com>.
solr-user，妳好
No, both. But first I have to face the indexing performance problem. Where can I see information about concurrent/parallel indexing on Solr? Thanks in advance.
----- Original Message -----
From: Toke Eskildsen
To: solr_user lucene_apache
Date: 2015-09-04, 00:57:51
Subject: Re: Re: Re: Re: concept and choice: custom sharding or auto sharding?
scott chu <sc...@udngroup.com> wrote:
?
> I keep forgeting to mention one thing along the discussion session.
> Our data is Chinese news articles and we use CJK tokenizer
> (i.e. 2-gram) currently. The time spent to indexing is quite slow,
> compared to indexing english articles. That's why I am so
> worrying about indexing performance on 10M Chinese docs
> and turn to study SolrCloud.
The performance problem is indexing and not searching? Solr supports concurrent indexing, so if you are able to send the data in parallel, just start as many indexing threads as you have cores. Of course that does not help if you are already doing that.
Also sanity check that you are not doing commits all the time.
- Toke Eskildsen
-----
???????????
??? AVG ?? - www.avg.com
??: 2015.0.6086 / ???: 4409/10567 - ????: 09/03/15
Re: Re: Re: Re: concept and choice: custom sharding or auto sharding?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
scott chu <sc...@udngroup.com> wrote:
> I keep forgeting to mention one thing along the discussion session.
> Our data is Chinese news articles and we use CJK tokenizer
> (i.e. 2-gram) currently. The time spent to indexing is quite slow,
> compared to indexing english articles. That's why I am so
> worrying about indexing performance on 10M Chinese docs
> and turn to study SolrCloud.
The performance problem is indexing and not searching? Solr supports concurrent indexing, so if you are able to send the data in parallel, just start as many indexing threads as you have cores. Of course that does not help if you are already doing that.
Also sanity check that you are not doing commits all the time.
- Toke Eskildsen
Re: Re: Re: Re: concept and choice: custom sharding or auto sharding?
Posted by scott chu <sc...@udngroup.com>.
solr-user，妳好
I keep forgeting to mention one thing along the discussion session. Our data is Chinese news articles and we use CJK tokenizer (i.e. 2-gram) currently. The time spent to indexing is quite slow, compared to indexing english articles. That's why I am so worrying about indexing performance on 10M Chinese docs and turn to study SolrCloud. It could also be the reason why we index 1M docs kinda slow. Frankly, we didn't delve into writing a better-performance Chinese tokenizer in past years due to some policy reason (However, we did make a plan to write one next year using MMSeg algorithm or 1-ngram+query-preprocessor).
----- Original Message -----
From: Erick Erickson
To: solr-user
Date: 2015-09-04, 00:07:43
Subject: Re: Re: Re: concept and choice: custom sharding or auto sharding?
bq: If you switch to SolrCloud, will you still keep numShards parameter to 1
yes. Although if you want to add more replicas you might want to specify that.
For 10M documents, I wouldn't be very fancy. Indexing them shouldn't take
very long, and I think your time would be better spent on other things than
trying to get fancy with splitshard and the like. Just create a
SolrCloud cluster
with as many replicas as you want and index from scratch unless it's
prohibitively expensive.
I can index 200M docs on my local Mac Pro in a couple of hours. Is it really
worth trying to do something you'll probably never do again (i.e. SPLITSHARD)?
If you really don't want to re-index _and_ you have only one shard in the
master/slave setup, here's what I'd do to migrate
1> create a new SolrCloud cluster with exactly one node (i.e. the "leader").
2> shut it down
3> copy the index from your master/slave to the new node, completely
replacing the data directory
4> bring the node back up and check it.
5> use the collecitons API ADDREPLICA command to bring up as many
replicas as you want, they'll pull down the index and from that point on
you should be good.
5a> In this case, it'll actually do a complete replication from the leader to
the followers, but thereafter incremental updates will be sent to all
the nodes in the cluster rather than the older style master/slave
occasional replication.
Best,
Erick
On Thu, Sep 3, 2015 at 8:54 AM, scott chu <sc...@udngroup.com> wrote:
>
> solr-user，妳好
>
> If you switch to SolrCloud, will you still keep numShards parameter to 1? If
> you are migrating to SolrCloud and going to split that single shard into
> multple shards, Wouldn't you have to reindex the data? Is it possible just
> put that single shard into SolrCloud and call SPLITSHARD API to split it?
>
> I ask this cause I'd like to try first use master-slave architecture, like
> Eric suggest that 10M is not a "vast" thing. Then later, I might migrate it
> to SolrCloud possibly because I want to take advange of the Zookeeper
> functionality for HA/DR.
>
> ----- Original Message -----
> From: Toke Eskildsen
> To: solr-user
> Date: 2015-09-03, 18:33:39
> Subject: Re: Re: concept and choice: custom sharding or auto sharding?
>
> On Thu, 2015-09-03 at 18:24 +0800, Scott Chu wrote:
>> Do you use master-slave or SolrCloud for that single shard?
>
> Due to legacy reasons we are just using 2 fully independent Solrs, each
> indexing independently, with an Apache load balancer in front for the
> searches. It does give us the occasional hiccup, so we'll be switching
> to SolrCloud at some point.
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6086 / 病毒庫: 4409/10567 - 發佈日期: 09/03/15
>
>
>
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6086 / 病毒庫: 4409/10567 - 發佈日期: 09/03/15
Re: Re: Re: concept and choice: custom sharding or auto sharding?
Posted by Erick Erickson <er...@gmail.com>.
bq: If you switch to SolrCloud, will you still keep numShards parameter to 1
yes. Although if you want to add more replicas you might want to specify that.
For 10M documents, I wouldn't be very fancy. Indexing them shouldn't take
very long, and I think your time would be better spent on other things than
trying to get fancy with splitshard and the like. Just create a
SolrCloud cluster
with as many replicas as you want and index from scratch unless it's
prohibitively expensive.
I can index 200M docs on my local Mac Pro in a couple of hours. Is it really
worth trying to do something you'll probably never do again (i.e. SPLITSHARD)?
If you really don't want to re-index _and_ you have only one shard in the
master/slave setup, here's what I'd do to migrate
1> create a new SolrCloud cluster with exactly one node (i.e. the "leader").
2> shut it down
3> copy the index from your master/slave to the new node, completely
replacing the data directory
4> bring the node back up and check it.
5> use the collecitons API ADDREPLICA command to bring up as many
replicas as you want, they'll pull down the index and from that point on
you should be good.
5a> In this case, it'll actually do a complete replication from the leader to
the followers, but thereafter incremental updates will be sent to all
the nodes in the cluster rather than the older style master/slave
occasional replication.
Best,
Erick
On Thu, Sep 3, 2015 at 8:54 AM, scott chu <sc...@udngroup.com> wrote:
>
> solr-user，妳好
>
> If you switch to SolrCloud, will you still keep numShards parameter to 1? If
> you are migrating to SolrCloud and going to split that single shard into
> multple shards, Wouldn't you have to reindex the data? Is it possible just
> put that single shard into SolrCloud and call SPLITSHARD API to split it?
>
> I ask this cause I'd like to try first use master-slave architecture, like
> Eric suggest that 10M is not a "vast" thing. Then later, I might migrate it
> to SolrCloud possibly because I want to take advange of the Zookeeper
> functionality for HA/DR.
>
> ----- Original Message -----
> From: Toke Eskildsen
> To: solr-user
> Date: 2015-09-03, 18:33:39
> Subject: Re: Re: concept and choice: custom sharding or auto sharding?
>
> On Thu, 2015-09-03 at 18:24 +0800, Scott Chu wrote:
>> Do you use master-slave or SolrCloud for that single shard?
>
> Due to legacy reasons we are just using 2 fully independent Solrs, each
> indexing independently, with an Apache load balancer in front for the
> searches. It does give us the occasional hiccup, so we'll be switching
> to SolrCloud at some point.
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6086 / 病毒庫: 4409/10567 - 發佈日期: 09/03/15
>
>
>
>
Re: Re: Re: concept and choice: custom sharding or auto sharding?
Posted by scott chu <sc...@udngroup.com>.
solr-user，妳好
If you switch to SolrCloud, will you still keep numShards parameter to 1? If you are migrating to SolrCloud and going to split that single shard into multple shards, Wouldn't you have to reindex the data? Is it possible just put that single shard into SolrCloud and call SPLITSHARD API to split it?
I ask this cause I'd like to try first use master-slave architecture, like Eric suggest that 10M is not a "vast" thing. Then later, I might migrate it to SolrCloud possibly because I want to take advange of the Zookeeper functionality for HA/DR.
----- Original Message -----
From: Toke Eskildsen
To: solr-user
Date: 2015-09-03, 18:33:39
Subject: Re: Re: concept and choice: custom sharding or auto sharding?
On Thu, 2015-09-03 at 18:24 +0800, Scott Chu wrote:
> Do you use master-slave or SolrCloud for that single shard?
Due to legacy reasons we are just using 2 fully independent Solrs, each
indexing independently, with an Apache load balancer in front for the
searches. It does give us the occasional hiccup, so we'll be switching
to SolrCloud at some point.
- Toke Eskildsen, State and University Library, Denmark
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6086 / 病毒庫: 4409/10567 - 發佈日期: 09/03/15
Re: Re: concept and choice: custom sharding or auto sharding?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2015-09-03 at 18:24 +0800, Scott Chu wrote:
> Do you use master-slave or SolrCloud for that single shard?
Due to legacy reasons we are just using 2 fully independent Solrs, each
indexing independently, with an Apache load balancer in front for the
searches. It does give us the occasional hiccup, so we'll be switching
to SolrCloud at some point.
- Toke Eskildsen, State and University Library, Denmark
Re: Re: concept and choice: custom sharding or auto sharding?
Posted by Scott Chu <sc...@udngroup.com>.
Do you use master-slave or SolrCloud for that single shard? Erick suggest that I can still can use SolrCloud for HA/DR purpose cause Zookeeper can do the work for me. Should I just give up master-slave choice even there's only one single shard?
Scott Chu，scott.chu@udngroup.com
2015/9/3
----- Original Message -----
From: Toke Eskildsen
To: solr-user
Date: 2015-09-03, 17:46:22
Subject: Re: Re: concept and choice: custom sharding or auto sharding?
On Wed, 2015-09-02 at 08:30 -0700, Erick Erickson wrote:
> Because I routinely see 50M docs on a single node and I've seen over 300M docs
> on a single node with sub-second responses.
For what it's worth, we also do article-based search of newspaper based
material (old OCR'ed papers). We use a single replicated shard for that
and it works fine (response times < 1s for 98.5% of the searches), with
faceting on 4 fields as well as grouping. There are 66M articles in a
340GB shard.
It is always hard to compare indexes, but I agree with Erick that having
performance problems with 10M documents calls for locating the
bottlenecks, before trying to scale the problem away.
- Toke Eskildsen, State and University Library, Denmark
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6086 / 病毒庫: 4409/10564 - 發佈日期: 09/02/15
Re: Re: concept and choice: custom sharding or auto sharding?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Wed, 2015-09-02 at 08:30 -0700, Erick Erickson wrote:
> Because I routinely see 50M docs on a single node and I've seen over 300M docs
> on a single node with sub-second responses.
For what it's worth, we also do article-based search of newspaper based
material (old OCR'ed papers). We use a single replicated shard for that
and it works fine (response times < 1s for 98.5% of the searches), with
faceting on 4 fields as well as grouping. There are 66M articles in a
340GB shard.
It is always hard to compare indexes, but I agree with Erick that having
performance problems with 10M documents calls for locating the
bottlenecks, before trying to scale the problem away.
- Toke Eskildsen, State and University Library, Denmark
Re: Re: concept and choice: custom sharding or auto sharding?
Posted by Erick Erickson <er...@gmail.com>.
bq: Why do you say: "at 10M documents there's rarely a need to shard at all?"
Because I routinely see 50M docs on a single node and I've seen over 300M docs
on a single node with sub-second responses. So if you're saying that
you see poor
performance at 1M docs then I suspect there's something radically
wrong with your
setup. Too little memory, very bad query patterns, whatever. If my
suspicion is true,
then sharding will just mask the underlying problem.
You need to quantify your performance concerns. It's one thing to say
"my node satisfies 50 queries-per-second with 500ms response time" and
another to say "My queries take 5,000 ms".
In the first case, you do indeed need to add more servers to increase QPS if
you need 500 QPS. And adding more slaves is the best way to do that.
In the second, you need to understand the slowdown because sharding
will be a band-aid.
This might help:
https://wiki.apache.org/solr/SolrPerformanceProblems
Best,
Erick
On Wed, Sep 2, 2015 at 8:19 AM, scott chu <sc...@udngroup.com> wrote:
>
> solr-user，妳好
>
> Do you mean I only have to put 10M documents in one index and copy it to
> many slaves in a classic Solr master-slave architecture to provide querying
> serivce on internet, and it won't have obvious downgrade of query
> performance? But I did have add 1M document into one index on master and
> provide 2 slaves to serve querying service on internet, the query
> performance is kinda sad. Why do you say: "at 10M documents there's rarely a
> need to shard at all?" Do I provide too few slaves? What amount of documents
> is suitable for a need for shard in SolrCloud?
>
> ----- Original Message -----
>
> From: Erick Erickson
> To: solr-user
> Date: 2015-09-02, 23:00:29
> Subject: Re: concept and choice: custom sharding or auto sharding?
>
> Frankly, at 10M documents there's rarely a need to shard at all.
> Why do you think you need to? This seems like adding
> complexity for no good reason. Sharding should only really
> be used when you have too many documents to fit on a single
> shard as it adds some overhead, restricts some
> possibilities (cross-core join for instance, a couple of
> grouping options don't work in distributed mode etc.).
>
> You can still run SolrCloud and have it manage multiple
> _replicas_ of a single shard for HA/DR.
>
> So this seems like an XY problem, you're asking specific
> questions about shard routing because you think it'll
> solve some problem without telling us what the problem
> is.
>
> Best,
> Erick
>
> On Wed, Sep 2, 2015 at 7:47 AM, scott chu <sc...@udngroup.com> wrote:
>> I post a question on Stackoverflow
>> http://stackoverflow.com/questions/32343813/custom-sharding-or-auto-sharding-on-solrcloud:
>> However, since this is a mail-list, I repost the question below to request
>> for suggestion and more subtle concept of SolrCloud's behavior on document
>> routing.
>> I want to establish a SolrCloud clsuter for over 10 millions of news
>> articles. After reading this article in Apache Solr Refernce guide: Shards
>> and Indexing Data in SolrCloud, I have a plan as follows:
>> Add prefix ED2001! to document ID where ED means some newspaper source and
>> 2001 is the year part in published date of news article, i.e. I want to put
>> all news articles of specific news paper source published in specific year
>> to a shard.
>> Create collection with router.name set to compositeID.
>> Add documents?
>> Query Collection?
>> Practically, I got some questions:
>> How to add doucments based on this plan? Do I have to specify special
>> parameters when updating the collection/core?
>> Is this called "custom sharding"? If not, what is "custom sharding"?
>> Is auto sharding a better choice for my case since there's a
>> shard-splitting feature for auto sharding when the shard is too big?
>> Can I query without _router_ parameter?
>> EDIT @ 2015/9/2:
>> This is how I think SolrCloud will do: "The amount of news articles of
>> specific newspaper source of specific year tends to be around a fix number,
>> e.g. Every year ED has around 80,000 articles, so each shard's size won't
>> increase dramatically. For the next year's news articles of ED, I only have
>> to add prefix 'ED2016!' to document ID, SolrCloud will create a new shard
>> for me (which contains all ED2016 articles), and later the Leader will
>> spread the replica of this new shard to other nodes (per replica per node
>> other than leader?)". Am I right? If yes, it seems no need for
>> shard-splitting.
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6086 / 病毒庫: 4409/10562 - 發佈日期: 09/02/15
>
>
>
>
>
Re: Re: concept and choice: custom sharding or auto sharding?
Posted by scott chu <sc...@udngroup.com>.
solr-user，妳好
Do you mean I only have to put 10M documents in one index and copy it to many slaves in a classic Solr master-slave architecture to provide querying serivce on internet, and it won't have obvious downgrade of query performance? But I did have add 1M document into one index on master and provide 2 slaves to serve querying service on internet, the query performance is kinda sad. Why do you say: "at 10M documents there's rarely a need to shard at all?" Do I provide too few slaves? What amount of documents is suitable for a need for shard in SolrCloud?
----- Original Message -----
From: Erick Erickson
To: solr-user
Date: 2015-09-02, 23:00:29
Subject: Re: concept and choice: custom sharding or auto sharding?
Frankly, at 10M documents there's rarely a need to shard at all.
Why do you think you need to? This seems like adding
complexity for no good reason. Sharding should only really
be used when you have too many documents to fit on a single
shard as it adds some overhead, restricts some
possibilities (cross-core join for instance, a couple of
grouping options don't work in distributed mode etc.).
You can still run SolrCloud and have it manage multiple
_replicas_ of a single shard for HA/DR.
So this seems like an XY problem, you're asking specific
questions about shard routing because you think it'll
solve some problem without telling us what the problem
is.
Best,
Erick
On Wed, Sep 2, 2015 at 7:47 AM, scott chu <sc...@udngroup.com> wrote:
> I post a question on Stackoverflow http://stackoverflow.com/questions/32343813/custom-sharding-or-auto-sharding-on-solrcloud:
> However, since this is a mail-list, I repost the question below to request for suggestion and more subtle concept of SolrCloud's behavior on document routing.
> I want to establish a SolrCloud clsuter for over 10 millions of news articles. After reading this article in Apache Solr Refernce guide: Shards and Indexing Data in SolrCloud, I have a plan as follows:
> Add prefix ED2001! to document ID where ED means some newspaper source and 2001 is the year part in published date of news article, i.e. I want to put all news articles of specific news paper source published in specific year to a shard.
> Create collection with router.name set to compositeID.
> Add documents?
> Query Collection?
> Practically, I got some questions:
> How to add doucments based on this plan? Do I have to specify special parameters when updating the collection/core?
> Is this called "custom sharding"? If not, what is "custom sharding"?
> Is auto sharding a better choice for my case since there's a shard-splitting feature for auto sharding when the shard is too big?
> Can I query without _router_ parameter?
> EDIT @ 2015/9/2:
> This is how I think SolrCloud will do: "The amount of news articles of specific newspaper source of specific year tends to be around a fix number, e.g. Every year ED has around 80,000 articles, so each shard's size won't increase dramatically. For the next year's news articles of ED, I only have to add prefix 'ED2016!' to document ID, SolrCloud will create a new shard for me (which contains all ED2016 articles), and later the Leader will spread the replica of this new shard to other nodes (per replica per node other than leader?)". Am I right? If yes, it seems no need for shard-splitting.
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6086 / 病毒庫: 4409/10562 - 發佈日期: 09/02/15
Re: concept and choice: custom sharding or auto sharding?
Posted by Erick Erickson <er...@gmail.com>.
Frankly, at 10M documents there's rarely a need to shard at all.
Why do you think you need to? This seems like adding
complexity for no good reason. Sharding should only really
be used when you have too many documents to fit on a single
shard as it adds some overhead, restricts some
possibilities (cross-core join for instance, a couple of
grouping options don't work in distributed mode etc.).
You can still run SolrCloud and have it manage multiple
_replicas_ of a single shard for HA/DR.
So this seems like an XY problem, you're asking specific
questions about shard routing because you think it'll
solve some problem without telling us what the problem
is.
Best,
Erick
On Wed, Sep 2, 2015 at 7:47 AM, scott chu <sc...@udngroup.com> wrote:
> I post a question on Stackoverflow http://stackoverflow.com/questions/32343813/custom-sharding-or-auto-sharding-on-solrcloud:
> However, since this is a mail-list, I repost the question below to request for suggestion and more subtle concept of SolrCloud's behavior on document routing.
> I want to establish a SolrCloud clsuter for over 10 millions of news articles. After reading this article in Apache Solr Refernce guide: Shards and Indexing Data in SolrCloud, I have a plan as follows:
> Add prefix ED2001! to document ID where ED means some newspaper source and 2001 is the year part in published date of news article, i.e. I want to put all news articles of specific news paper source published in specific year to a shard.
> Create collection with router.name set to compositeID.
> Add documents?
> Query Collection?
> Practically, I got some questions:
> How to add doucments based on this plan? Do I have to specify special parameters when updating the collection/core?
> Is this called "custom sharding"? If not, what is "custom sharding"?
> Is auto sharding a better choice for my case since there's a shard-splitting feature for auto sharding when the shard is too big?
> Can I query without _router_ parameter?
> EDIT @ 2015/9/2:
> This is how I think SolrCloud will do: "The amount of news articles of specific newspaper source of specific year tends to be around a fix number, e.g. Every year ED has around 80,000 articles, so each shard's size won't increase dramatically. For the next year's news articles of ED, I only have to add prefix 'ED2016!' to document ID, SolrCloud will create a new shard for me (which contains all ED2016 articles), and later the Leader will spread the replica of this new shard to other nodes (per replica per node other than leader?)". Am I right? If yes, it seems no need for shard-splitting.
Re: concept and choice: custom sharding or auto sharding?
Posted by Charlie Hull <ch...@flax.co.uk>.
On 02/09/2015 15:47, scott chu wrote:
> I post a question on Stackoverflow http://stackoverflow.com/questions/32343813/custom-sharding-or-auto-sharding-on-solrcloud:
> However, since this is a mail-list, I repost the question below to request for suggestion and more subtle concept of SolrCloud's behavior on document routing.
> I want to establish a SolrCloud clsuter for over 10 millions of news articles. After reading this article in Apache Solr Refernce guide: Shards and Indexing Data in SolrCloud, I have a plan as follows:
> Add prefix ED2001! to document ID where ED means some newspaper source and 2001 is the year part in published date of news article, i.e. I want to put all news articles of specific news paper source published in specific year to a shard.
> Create collection with router.name set to compositeID.
> Add documents?
> Query Collection?
> Practically, I got some questions:
> How to add doucments based on this plan? Do I have to specify special parameters when updating the collection/core?
> Is this called "custom sharding"? If not, what is "custom sharding"?
> Is auto sharding a better choice for my case since there's a shard-splitting feature for auto sharding when the shard is too big?
> Can I query without _router_ parameter?
> EDIT @ 2015/9/2:
> This is how I think SolrCloud will do: "The amount of news articles of specific newspaper source of specific year tends to be around a fix number, e.g. Every year ED has around 80,000 articles, so each shard's size won't increase dramatically. For the next year's news articles of ED, I only have to add prefix 'ED2016!' to document ID, SolrCloud will create a new shard for me (which contains all ED2016 articles), and later the Leader will spread the replica of this new shard to other nodes (per replica per node other than leader?)". Am I right? If yes, it seems no need for shard-splitting.
>
Think about your query pattern when you decide how to shard. If most of
your queries are for recent articles, then some shards will be loaded
far more than others. Here's a rather old blog post we wrote on the
subject (actually based on Xapian, another open source search engine,
but the concepts are the same for Solr):
http://www.flax.co.uk/blog/2009/04/
Cheers
Charlie
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
concept and choice: custom sharding or auto sharding?
Posted by scott chu <sc...@udngroup.com>.
I post a question on Stackoverflow http://stackoverflow.com/questions/32343813/custom-sharding-or-auto-sharding-on-solrcloud:
However, since this is a mail-list, I repost the question below to request for suggestion and more subtle concept of SolrCloud's behavior on document routing.
I want to establish a SolrCloud clsuter for over 10 millions of news articles. After reading this article in Apache Solr Refernce guide: Shards and Indexing Data in SolrCloud, I have a plan as follows:
Add prefix ED2001! to document ID where ED means some newspaper source and 2001 is the year part in published date of news article, i.e. I want to put all news articles of specific news paper source published in specific year to a shard.
Create collection with router.name set to compositeID.
Add documents?
Query Collection?
Practically, I got some questions:
How to add doucments based on this plan? Do I have to specify special parameters when updating the collection/core?
Is this called "custom sharding"? If not, what is "custom sharding"?
Is auto sharding a better choice for my case since there's a shard-splitting feature for auto sharding when the shard is too big?
Can I query without _router_ parameter?
EDIT @ 2015/9/2:
This is how I think SolrCloud will do: "The amount of news articles of specific newspaper source of specific year tends to be around a fix number, e.g. Every year ED has around 80,000 articles, so each shard's size won't increase dramatically. For the next year's news articles of ED, I only have to add prefix 'ED2016!' to document ID, SolrCloud will create a new shard for me (which contains all ED2016 articles), and later the Leader will spread the replica of this new shard to other nodes (per replica per node other than leader?)". Am I right? If yes, it seems no need for shard-splitting.
Re: Strange behavior of solr
Posted by Erik Hatcher <er...@gmail.com>.
See example/films/README.txt
The “name” field is guessed incorrectly (because the first film has name=“.45”, so indexing errors once it hits a name value that is no longer numeric. The README provides a command to define the name field *before* indexing. If you’ve indexed and had the name field guessed incorrectly and created, you’ll need to delete and recreate the collection, then define the name field, then reindex.
We used to have a fake film at the top to allow field guessing to “work”, but I felt that was too fake and that the example should be true to what happens with real world data and the pitfalls of allowing field type guessing to guess incorrectly.
—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com
> On Sep 2, 2015, at 5:17 AM, Long Yan <lo...@robotron.de> wrote:
>
> Hey,
> I have created a core with
> bin\solr create -c mycore
>
> I want to index the csv sample files from solr-5.2.1
>
> If I index film.csv under solr-5.2.1\example\films\, solr can only index this file until the line
> "2046,Wong Kar-wai,Romance Film|Fantasy|Science Fiction|Drama,,/en/2046_2004,2004-05-20"
>
> But if I at first index books.csv under solr-5.2.1\example\exampledocs and then index film.csv, solr can index all lines in film.csv
>
> Why?
>
> Regards
> Long Yan
>
>
Re: Strange behavior of solr
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Is there any error message in the log when Solr stops indexing the file at
line 2046?
Regards,
Edwin
On 2 September 2015 at 17:17, Long Yan <lo...@robotron.de> wrote:
> Hey,
> I have created a core with
> bin\solr create -c mycore
>
> I want to index the csv sample files from solr-5.2.1
>
> If I index film.csv under solr-5.2.1\example\films\, solr can only index
> this file until the line
> "2046,Wong Kar-wai,Romance Film|Fantasy|Science
> Fiction|Drama,,/en/2046_2004,2004-05-20"
>
> But if I at first index books.csv under solr-5.2.1\example\exampledocs and
> then index film.csv, solr can index all lines in film.csv
>
> Why?
>
> Regards
> Long Yan
>
>
>