You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "scott.chu" <sc...@udngroup.com> on 2016/05/11 03:11:13 UTC
[scottchu] What kind of configuration to use for this size of news
data?
I want to build a Solr engine for over 60-year news articles. My requests are (I use Solr 5.4.1):
1> Currently over 10M no. of docs.
2> Currently over 60GB total data size.
3> The no. of docs and data size will keep growing at the rate of 1000 no. of docs(or 8MB size) per day.
4> There are totally 5-6 different newspaper types.
My questions are:
1> Is it wokable enough just to use master-slave model? Or should I turn of SolrCloud? (I ask this due to our system management group never manage a distributed system before and they also have no knowedge of Zookeeper, shards, etc. Also they don't know how to backup/restore distributed data.)
2> Say if I choose Solrcloud anyway. I wish to keep one shard owning one specific year of data. Can it be done? What configuration should I do? (AFAIK, SolrCloud distributes data based on some intrinsic routing algorithm.)
3> If I wish to create another Solr engine with one or two particular paper types. Is it possible to copy their data directly from the big central Solr engine? Or I have to rebuild index from raw articles data? (Our business has this possibility of needs.)
I'd like to hear and use some well suggestion and experiences.
Thanks in advance and best regards.
scott.chu，scott.chu@udngroup.com
2016/5/11 (週三)
Re: [scottchu] What kind of configuration to use for this size
ofnews data?
Posted by "scott.chu" <sc...@udngroup.com>.
Hi, Charlie,
Thanks first for your concrete answer. I have further questions as written in blue color below.
scott.chu，scott.chu@udngroup.com
2016/5/11 (週三)
----- Original Message -----
From: Charlie Hull
To: solr-user@lucene.apache.org
CC:
Date: 2016/5/11 (週三) 16:21
Subject: Re: [scottchu] What kind of configuration to use for this size ofnews data?
On 11/05/2016 04:27, scott.chu wrote:
> Fix some typos, add some words and resend same question =>
>
> I want to build a Solr engine for over 60-year news articles. My
> requests are (I use Solr 5.4.1):
Hi Scott,
We've actually done something very similar for the our client NLA Media
Access in the UK, who handle licensing of most UK newspaper content.
They have over 45m docs going back to 2006.
>
> 1> Currently over 10M no. of docs. 2> Currently over 60GB total data
> size. 3> The no. of docs and data size will keep growing at the rate
> of 1000 no. of docs(or 8MB size) per day. 4> There are totally 5-6
> different newspaper types.
>
> My questions are: 1> Is it wokable enough just to use master-slave
> model? Or should I turn to SolrCloud? (I ask this due to our system
> management group never manage a distributed system before and they
> also have no knowedge of Zookeeper, shards, etc. Also they don't know
> how to backup/restore distributed data.)
Workable yes, advisable no. You should get much better reliability &
performance with SolrCloud once it's set up. Also, if you have
replication set up correctly the need for backup/restore will be
significantly reduced and may be unnecessary.
We used master-slave for News UK's Solr setup (articles from The Times
and other papers) but this was before SolrCloud had properly arrived.
We'd only use master-slave rarely now.
If I use SolrCloud, I know I have to setup Zookeeper. I know there're something called 'quorum' or 'ensemble' in Zookeeper terminologies. I also know there is a need for (2n+1) Zookeeper nodes per n SolrCloud nodes. Is your case running one SolrCloud node per one machine (Whether PM or VM). According to your experiences, how many nodes , including SolrCloud's and Zookeeper's, do I need to setup? Is Replication in SolrCloud easy to setup as that in old version? (I setup replication solrconfig.xml and use solrcore.properties file to setup/switch roles in Solr node, rather than defining role directly in solrconfig.xml)
> 2> Say if I choose Solrcloud anyway. I wish to keep one shard owning
> one specific year of data. Can it be done?
Yes it can, but it may not be a good idea. If a large proportion of your
queries hit recent news you may find one shard dealing with more queries
than the others and becoming overloaded. Here's a blog post we wrote a
long time ago about this - ignore the name Xapian, this applies to Solr
as well:
http://www.flax.co.uk/blog/2009/04/25/distributed-search-and-partition-functions/
What configuration should
> I do? (AFAIK, SolrCloud distributes data based on some intrinsic
> routing algorithm.)
You can choose how to route data at indexing time:
https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud
You are right. I do neglect this condition. I'll think twice and could drop out my idea. Thanks for sharing the blog article. I'll take a good look at it.
>3> If I wish to create another Solr engine with
> one or two particular paper types. Is it possible to copy their index
> data directly from the big central Solr engine? Or I have to rebuild
> index from raw articles data? (Our business has this possibility of
> needs.)
Yes, I guess so, but why copy it when you could just search it with a
filter for the paper types?
We have a special biz case called 'buyout newspaper search service'. Customers buy intranet license to use search service for articles of some newspaper types and some range of publish dates, e.g. paper type 'A' for 2010-2012 and paper type 'B' for 2015. The buyout means we have to install who search service at customer site and customer can only use search service within their enterprise intranet environment. So you know, I have to build a special Solr server for each of such customers. Your idea of filtering is very much like ElasticSearch's multitenancy, which both are not fit in our buyout biz model. Do you have any suggestion for building Solr server in such condition?
>
> I'd like to hear and use some well suggestion and experiences.
>
> Thanks in advance and best regards.
>
> Scott Chu @ 2016/5/11 11:26 GMT+8
>
Hope this helps!
Cheers
Charlie
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6189 / 病毒庫: 4568/12206 - 發佈日期: 05/10/16
Re: [scottchu] What kind of configuration to use for this size of
news data?
Posted by Charlie Hull <ch...@flax.co.uk>.
On 11/05/2016 04:27, scott.chu wrote:
> Fix some typos, add some words and resend same question =>
>
> I want to build a Solr engine for over 60-year news articles. My
> requests are (I use Solr 5.4.1):
Hi Scott,
We've actually done something very similar for the our client NLA Media
Access in the UK, who handle licensing of most UK newspaper content.
They have over 45m docs going back to 2006.
>
> 1> Currently over 10M no. of docs. 2> Currently over 60GB total data
> size. 3> The no. of docs and data size will keep growing at the rate
> of 1000 no. of docs(or 8MB size) per day. 4> There are totally 5-6
> different newspaper types.
>
> My questions are: 1> Is it wokable enough just to use master-slave
> model? Or should I turn to SolrCloud? (I ask this due to our system
> management group never manage a distributed system before and they
> also have no knowedge of Zookeeper, shards, etc. Also they don't know
> how to backup/restore distributed data.)
Workable yes, advisable no. You should get much better reliability &
performance with SolrCloud once it's set up. Also, if you have
replication set up correctly the need for backup/restore will be
significantly reduced and may be unnecessary.
We used master-slave for News UK's Solr setup (articles from The Times
and other papers) but this was before SolrCloud had properly arrived.
We'd only use master-slave rarely now.
> 2> Say if I choose Solrcloud anyway. I wish to keep one shard owning
> one specific year of data. Can it be done?
Yes it can, but it may not be a good idea. If a large proportion of your
queries hit recent news you may find one shard dealing with more queries
than the others and becoming overloaded. Here's a blog post we wrote a
long time ago about this - ignore the name Xapian, this applies to Solr
as well:
http://www.flax.co.uk/blog/2009/04/25/distributed-search-and-partition-functions/
What configuration should
> I do? (AFAIK, SolrCloud distributes data based on some intrinsic
> routing algorithm.)
You can choose how to route data at indexing time:
https://cwiki.apache.org/confluence/display/solr/Shards+and+Indexing+Data+in+SolrCloud
>3> If I wish to create another Solr engine with
> one or two particular paper types. Is it possible to copy their index
> data directly from the big central Solr engine? Or I have to rebuild
> index from raw articles data? (Our business has this possibility of
> needs.)
Yes, I guess so, but why copy it when you could just search it with a
filter for the paper types?
>
> I'd like to hear and use some well suggestion and experiences.
>
> Thanks in advance and best regards.
>
> Scott Chu @ 2016/5/11 11:26 GMT+8
>
Hope this helps!
Cheers
Charlie
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
Re(2): [scottchu] What kind of configuration to use for this size
ofnews data?
Posted by "scott.chu" <sc...@udngroup.com>.
Hi Toke and Charile,
Thanks for sharing your cases and your well suggestions. After reading through your mails, I'll delve into SolrCloud. One thing I'd like to share with all of people in maillist: Chinese corpus could create a dramatically large size of index with respect to what tokenization method we use. CJK tokenizer falls into this situation and synonym becomes very gard to establish (For every Chinese words composed by n Chinese character, I have to set C(n,2) synonym items, which is impossible for me to do it). Unfortunately, my old Solr 3.5 server uses it.
This time I have two choices for Chinese tokenizing:
1> Algorithmic way: "Using standard tokenizer + query with quoting Chinese word as phrase (by quoting with double quotes). This maintains a fairly size of index and carry out the effect similar with Google's search. This also makes synonym be feasible practically.
2> Dictionary-oriented way: I install mmseg4j (The theory is from Taiwan but carried out by China). But the problem is on how to maintain an up-to-date dictionary, especially for "news". News may create never-before new nouns as the times goes by.
You both have experiences installing a newspaper search site. I'd like to know what way of tokenizer you use and how do you maintain an up-to-date dictionary assume if you use the 2nd way. I know most of the Solr tokenizers in China uses the 2nd way, so I'm curious how they maintain an up-to-date dictionary. If there's someone has experiences running a Chinese-corpus Solr server, I'll appreciate if you're willing to share your case.
Thanks again and best regards, you guys save many time for my job ^_^
scott.chu，scott.chu@udngroup.com
2016/5/12 (週四)
----- Original Message -----
From: Toke Eskildsen
To: solr-user ; scott(自己)
CC:
Date: 2016/5/11 (週三) 18:55
Subject: Re: [scottchu] What kind of configuration to use for this size ofnews data?
On Wed, 2016-05-11 at 11:27 +0800, scott.chu wrote:
> I want to build a Solr engine for over 60-year news articles. My
> requests are (I use Solr 5.4.1):
Charlie Hull has given you an fine answer, which I agree with fully, so
I'll just add a bit from our experience.
We are running a similar service for Danish newspapers. We have 16M
OCR'ed pages, split into 250M+ articles, for 1.4TB total index size.
Everything in a single shard on a 64GB machine with SSDs.
We do faceting, range faceting and grouping as part of basic search.
That works okay (sub-second response times) for the bulk of our
requests, but when the hitCount gets above 10M, performance gets poor.
For the real heavy hitters, basically matching everything, we encounter
20 second response times.
This is not acceptable, so we will be switching to SolrCloud and
multiple shards (on the same machine, as our bottleneck is single
CPU-core performance). However, you have a smaller corpus and the growth
rate does not look alarming.
Putting all this together, I would advice you to try and put everything
in a single shard to avoid the overhead of distributed search. If that
performs well enough for single queries, then add replicas with
SolrCloud to get redundancy and scale throughput. Should you need to
shard at a later time, this will be easy with SolrCloud.
- Toke Eskildsen, State and University Library, Denmark
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6189 / 病毒庫: 4568/12206 - 發佈日期: 05/10/16
Re: [scottchu] What kind of configuration to use for this size of
news data?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Wed, 2016-05-11 at 11:27 +0800, scott.chu wrote:
> I want to build a Solr engine for over 60-year news articles. My
> requests are (I use Solr 5.4.1):
Charlie Hull has given you an fine answer, which I agree with fully, so
I'll just add a bit from our experience.
We are running a similar service for Danish newspapers. We have 16M
OCR'ed pages, split into 250M+ articles, for 1.4TB total index size.
Everything in a single shard on a 64GB machine with SSDs.
We do faceting, range faceting and grouping as part of basic search.
That works okay (sub-second response times) for the bulk of our
requests, but when the hitCount gets above 10M, performance gets poor.
For the real heavy hitters, basically matching everything, we encounter
20 second response times.
This is not acceptable, so we will be switching to SolrCloud and
multiple shards (on the same machine, as our bottleneck is single
CPU-core performance). However, you have a smaller corpus and the growth
rate does not look alarming.
Putting all this together, I would advice you to try and put everything
in a single shard to avoid the overhead of distributed search. If that
performs well enough for single queries, then add replicas with
SolrCloud to get redundancy and scale throughput. Should you need to
shard at a later time, this will be easy with SolrCloud.
- Toke Eskildsen, State and University Library, Denmark
[scottchu] What kind of configuration to use for this size of news
data?
Posted by "scott.chu" <sc...@udngroup.com>.
Fix some typos, add some words and resend same question =>
I want to build a Solr engine for over 60-year news articles. My requests are (I use Solr 5.4.1):
1> Currently over 10M no. of docs.
2> Currently over 60GB total data size.
3> The no. of docs and data size will keep growing at the rate of 1000 no. of docs(or 8MB size) per day.
4> There are totally 5-6 different newspaper types.
My questions are:
1> Is it wokable enough just to use master-slave model? Or should I turn to SolrCloud? (I ask this due to our system management group never manage a distributed system before and they also have no knowedge of Zookeeper, shards, etc. Also they don't know how to backup/restore distributed data.)
2> Say if I choose Solrcloud anyway. I wish to keep one shard owning one specific year of data. Can it be done? What configuration should I do? (AFAIK, SolrCloud distributes data based on some intrinsic routing algorithm.)
3> If I wish to create another Solr engine with one or two particular paper types. Is it possible to copy their index data directly from the big central Solr engine? Or I have to rebuild index from raw articles data? (Our business has this possibility of needs.)
I'd like to hear and use some well suggestion and experiences.
Thanks in advance and best regards.
Scott Chu @ 2016/5/11 11:26 GMT+8
Re(2): Import html data in mysql and map schemas using
onlySolrCELL+TIKA+DIH [scottchu]
Posted by "scott.chu" <sc...@udngroup.com>.
I read that but can't quite understand the steps! That's why I ask help here.
scott.chu，scott.chu@udngroup.com
2016/5/24 (週二)
----- Original Message -----
From: Markus Jelsma
To: solr-user ; solr-user
CC:
Date: 2016/5/24 (週二) 17:52
Subject: RE: Import html data in mysql and map schemas using onlySolrCELL+TIKA+DIH [scottchu]
Hello - did you find this manual page? It explains how HTML can be uploaded.
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Solr+Cell+using+Apache+Tika
Markus
-----Original message-----
> From:scott.chu <sc...@udngroup.com>
> Sent: Tuesday 24th May 2016 7:48
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Import html data in mysql and map schemas using only SolrCELL+TIKA+DIH [scottchu]
>
> Can anyone show me an example or short help of how I can do it? I am to use Solr 5 or up to carry out it.
>
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/24 (週二)
> ----- Original Message -----
> From: scott(自己)
> To: solr-user
> CC:
> Date: 2016/5/20 (週五) 14:17
> Subject: Import html data in mysql and map schemas using only SolrCELL+TIKA+DIH [scottchu]
>
>
>
> I have a mysql table with over 300M blog articles. The records are in html format. Is it possible to import these records using only Solr CELL+TIKA+DIH to some Solr with schema? I mean when importing, I can map schema on mysql to schema in Solr?
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/20 (週五)
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12262 - 發佈日期: 05/19/16
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12285 - 發佈日期: 05/23/16
RE: Import html data in mysql and map schemas using only
SolrCELL+TIKA+DIH [scottchu]
Posted by Markus Jelsma <ma...@openindex.io>.
Hello - did you find this manual page? It explains how HTML can be uploaded.
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Solr+Cell+using+Apache+Tika
Markus
-----Original message-----
> From:scott.chu <sc...@udngroup.com>
> Sent: Tuesday 24th May 2016 7:48
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Import html data in mysql and map schemas using only SolrCELL+TIKA+DIH [scottchu]
>
> Can anyone show me an example or short help of how I can do it? I am to use Solr 5 or up to carry out it.
>
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/24 (週二)
> ----- Original Message -----
> From: scott(自己)
> To: solr-user
> CC:
> Date: 2016/5/20 (週五) 14:17
> Subject: Import html data in mysql and map schemas using only SolrCELL+TIKA+DIH [scottchu]
>
>
>
> I have a mysql table with over 300M blog articles. The records are in html format. Is it possible to import these records using only Solr CELL+TIKA+DIH to some Solr with schema? I mean when importing, I can map schema on mysql to schema in Solr?
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/20 (週五)
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12262 - 發佈日期: 05/19/16
>
Re: Updating error while add doc to Solrcloud
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/17/2016 12:29 AM, scott.chu wrote:
> I build Solrcloud with 2 nodes, 1 shard, 2 replica. I add doc in xml format using post.jar up to 2.85M+ no. of docs and 10gb index size. When I add more docs. the solr.log shows:
>
> --------------------------------------
> 2016-05-17 14:01:09,024 WARN (main) [ ] o.e.j.s.h.RequestLogHandler !RequestLog
> 2016-05-17 14:01:09,275 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@57fffcd7{/solr,file:/D:/portable_sw/solr-5.4.1/server/solr-webapp/webapp/,STARTING}{D:\portable_sw\solr-5.4.1\server/solr-webapp/webapp} has uncovered http methods for path: /
> 2016-05-17 14:01:09,346 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from D:\portable_sw\solr-5.4.1\mynodes\cloud\node1\lib to classpath: D:\portable_sw\solr-5.4.1\mynodes\cloud\node1\lib
> 2016-05-17 14:01:11,419 WARN (coreLoadExecutor-7-thread-1-processing-n:10.18.59.179:8983_solr) [c:cugna s:shard1 r:core_node2 x:cugna_shard1_replica2] o.a.s.u.UpdateLog Exception reverse reading log
> java.io.EOFException
> ...
> --------------------------------------
I have no idea what those warnings are saying. The first two are from
Jetty.
I assume that your xml is in the Solr XML update format. How big is the
xml file you are sending? Solr has a default 2MB limit on the POST body
size. This can be increased with the formdataUploadLimitInKB parameter
... but do not increase it too much. One of the reason that the limits
are there to keep memory usage down. Instead of trying to index one
really massive file, break it into pieces small enough to fit in the
POST size limit.
> Later I stop all and kill write.lock (I ususally do this in Solr 3 when add doc fails) and add doc again but Solrcloud show can't find write.lock. So I recover write.lock and call post.jar again. The output shows:
If you need to delete write.lock after you stop Solr, that means you are
forcibly terminating Solr, not asking it to stop and waiting for
graceful shutdown. When doing a normal shutdown, Solr will erase all
the write.lock files itself. The bin/solr script only waits five
seconds for Solr to stop before proceeding with forcible termination,
which often isn't enough, and should be adjusted.
Replying to things stated later in the thread:
If you only supply Solr with one zookeeper host, then Solr will stop
working if that zookeeper host goes down, because it will not have any
information about the other servers. Zookeeper servers do not currently
inform zookeeper clients about other servers in the ensemble. When
Zookeeper 3.5 comes out and Solr updates to use that version, this
*might* work, but I am not sure. The best option is to simply inform
Solr about all your zookeeper servers when you start it, particularly
because you may not be sure which of those servers are actually *up*
when you start.
Zookeeper 3.5 will support dynamic cluster membership -- adding and
removing servers without restarting servers or clients. You should
still give Solr at least three servers when you start it, though -- so
it can reliably *find* the cluster.
This might be info you know already: Running everything on one server
is NOT suitable for production. Once you do get to production with a
minimum of three zookeeper machines and two Solr machines (which may
also be zookeeper machines), if you do end up needing to run multiple
Solr instances per machine, this is the recommended way to do it (on
Linux and other *NIX operating systems):
https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production#TakingSolrtoProduction-RunningmultipleSolrnodesperhost
Thanks,
Shawn
Re: Updating error while add doc to Solrcloud
Posted by Erick Erickson <er...@gmail.com>.
I _think_ you are using "schemaless" mode and the
issue is that Solr guesses the type of the field based
on the first doc it encounters. Thereafter, if any
incoming doc has a different field (say the "guess"
is an int type and later something that's not an
int is in that field) then it is rejected. This is somewhat
borne out by the fact that when you blow away the collection,
the problem doc indexes.
By default, modern Solr uses "managed schema", which is
different than, but related to "schemaless". What I'd do:
Go to managed schema or even "classic" schema, here's
a place to get you started:
https://cwiki.apache.org/confluence/display/solr/Schema+Factory+Definition+in+SolrConfig
Best,
Erick
On Mon, May 16, 2016 at 11:36 PM, scott.chu <sc...@udngroup.com> wrote:
>
> I clear the cugna collection data (by renaming 'data' folder to 'xdata')and restart Solrcloud. I add previous possible-error xml doc, it succeeds. So I'm sure doc data has no problem. Is it because the index file size is too large? If the zk nodes fails during adding doc, could it cause this updating error since I do see some "can't find leader error' in solr log?
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/17 (週二)
> ----- Original Message -----
> From: scott(自己)
> To: solr-user
> CC:
> Date: 2016/5/17 (週二) 14:29
> Subject: Updating error while add doc to Solrcloud
>
>
>
> I build Solrcloud with 2 nodes, 1 shard, 2 replica. I add doc in xml format using post.jar up to 2.85M+ no. of docs and 10gb index size. When I add more docs. the solr.log shows:
>
> --------------------------------------
> 2016-05-17 14:01:09,024 WARN (main) [ ] o.e.j.s.h.RequestLogHandler !RequestLog
> 2016-05-17 14:01:09,275 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@57fffcd7{/solr,file:/D:/portable_sw/solr-5.4.1/server/solr-webapp/webapp/,STARTING}{D:\portable_sw\solr-5.4.1\server/solr-webapp/webapp} has uncovered http methods for path: /
> 2016-05-17 14:01:09,346 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from D:\portable_sw\solr-5.4.1\mynodes\cloud\node1\lib to classpath: D:\portable_sw\solr-5.4.1\mynodes\cloud\node1\lib
> 2016-05-17 14:01:11,419 WARN (coreLoadExecutor-7-thread-1-processing-n:10.18.59.179:8983_solr) [c:cugna s:shard1 r:core_node2 x:cugna_shard1_replica2] o.a.s.u.UpdateLog Exception reverse reading log
> java.io.EOFException
> ...
> --------------------------------------
>
> Later I stop all and kill write.lock (I ususally do this in Solr 3 when add doc fails) and add doc again but Solrcloud show can't find write.lock. So I recover write.lock and call post.jar again. The output shows:
>
> --------------------------------------
> "SimplePostTool version 5.0.0
> Posting files to [base] url http://localhost:8983/solr/cugna/update using content-type application/xml...
> POSTing file NMLBOym_a_UN2000_10_20160511_1014.xml to [base]
> SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url: http://localhost:8983/solr/cugna/update
> SimplePostTool: WARNING: Response: <?xml version="1.0" encoding="UTF-8"?>
> <response>
> <lst name="responseHeader"><int name="status">400</int><int name="QTime">3</int></lst><lst name="error"><str name="msg">
> Exception writing document id un_555917 to the index; possible analysis error.</str><int name="code">400</int></lst>
> </response>
> SimplePostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 40
> 0 for URL: http://localhost:8983/solr/cugna/update
> 1 files indexed.
> COMMITting Solr index changes to http://localhost:8983/solr/cugna/update...
> Time spent: 0:00:00.259"
> -------------------------------------
>
> I thought it's that doc un_555917 problem, then I comment out it in xml and do again, it keeps showing same error to every single doc. I assume there's something wrong with Solrcloud. Does anyone experience this before? What could be the problem? Can I recover it? Or I have to add all docs again?
>
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/17 (週二)
> ----- Original Message -----
> From: scott(自己)
> To: solr-user
> CC:
> Date: 2016/5/17 (週二) 09:39
> Subject: Re(2): [scottchu] Cab I migrate solrcloud by just copying wholepackagefolder?
>
>
>
> OK! Thanks for reminding. I'll stick to the convention.
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/17 (週二)
> ----- Original Message -----
> From: Chris Hostetter
> To: solr-user ; scott(自己)
> CC:
> Date: 2016/5/17 (週二) 02:43
> Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole packagefolder?
>
>
>
> : Message-Id: <7F...@udngroup.com>
> : Subject: [scottchu] Cab I migrate solrcloud by just copying whole package
> : folder?
> : References:
> : <DC...@udngroup.com><C162E1B1F0311FC46A64E2E2AA
> : 729138@udngroup.com>,
> : <85...@elyograg.org>
> : In-Reply-To: <85...@elyograg.org>
>
> https://people.apache.org/~hossman/#threadhijack
> Thread Hijacking on Mailing Lists
>
> When starting a new discussion on a mailing list, please do not reply to
> an existing message, instead start a fresh email. Even if you change the
> subject line of your email, other mail headers still track which thread
> you replied to and your question is "hidden" in that thread and gets less
> attention. It makes following discussions in the mailing list archives
> particularly difficult.
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
>
> -----
> ???????????
> ??? AVG ?? - www.avg.com
> ??: 2015.0.6201 / ???: 4568/12238 - ????: 05/15/16
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12245 - 發佈日期: 05/16/16
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12245 - 發佈日期: 05/16/16
Re: [scottchu] How to specify multiple zk nodes using solr start
command under Windows
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
it's roundabout, but this might work -- ask for the healthcheck status
(from the solr box) and hit each zkNode separately.
I'm on Linux so you'll have to translate to Windows... using the solr.cmd
file I assume...
./solr healthcheck -z 192.168.56.5:2181/solr5_4 -c collectionName
./solr healthcheck -z 192.168.56.6:2181/solr5_4 -c collectionName
./solr healthcheck -z 192.168.56.7:2181/solr5_4 -c collectionName
My original command included all the IP addresses without port numbers.
This does return healthcheck info (the same info) when I do it for each
zkNode separately...
I assume that if they've all got info for all your replicas and/or shards,
they're working as an ensemble.
You could also go looking at each zkNode (using the zkCli tool) and verify
that your collection appears where you expect it.
On Tue, May 17, 2016 at 10:17 AM, Abdel Belkasri <be...@gmail.com> wrote:
> The repetition is just a cut and paste from Scott's post.
>
> How can I check if I am getting the ensemble or just a single zk?
>
> Also if this is not the way to specify an ensemble, what is the right way?
>
> Because the comma delimited list does not work, I concur with Scott.
>
> On Tue, May 17, 2016 at 11:49 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
> > Are you absolutely sure you're getting an _ensemble_ and
> > not just connecting to a single node? My suspicion (without
> > proof) is that you're just getting one -z option. It'll work as
> > long as that ZK instance stays up, but it won't be fault-tolerant.
> >
> > And again you repeated the port (2181) twice.
> >
> > Best,
> > Erick
> >
> > On Tue, May 17, 2016 at 8:02 AM, Abdel Belkasri <be...@gmail.com>
> > wrote:
> > > Hi Scott,
> > > what worked for me in Windows is this (no ",")
> > > bin\Solr start -c -s mynodes\node1 -z localhost:2181 -z localhost:2181
> -z
> > > localhost:2183
> > >
> > > -- Hope this helps
> > > Abdel.
> > >
> > > On Tue, May 17, 2016 at 3:35 AM, scott.chu <sc...@udngroup.com>
> > wrote:
> > >
> > >> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
> > >> Go into Solr 5.4.1 root folder and issue and issue the command in
> > article
> > >> 'Setting Up an External ZooKeeper Ensemble' in reference guide
> > >>
> > >> bin\Solr start -c -s mynodes\node1 -z
> > >> localhost:2181,localhost:2181,localhost:2183
> > >>
> > >> but it doesn't run but just show help page of start command in
> solr.cmd.
> > >> How should I issue the correct command?
> > >>
> > >
> > >
> > >
> > > --
> > > Abdel K. Belkasri, PhD
> >
>
>
>
> --
> Abdel K. Belkasri, PhD
>
Re: Import html data in mysql and map schemas using only
SolrCELL+TIKA+DIH [scottchu]
Posted by "scott.chu" <sc...@udngroup.com>.
Can anyone show me an example or short help of how I can do it? I am to use Solr 5 or up to carry out it.
scott.chu，scott.chu@udngroup.com
2016/5/24 (週二)
----- Original Message -----
From: scott(自己)
To: solr-user
CC:
Date: 2016/5/20 (週五) 14:17
Subject: Import html data in mysql and map schemas using only SolrCELL+TIKA+DIH [scottchu]
I have a mysql table with over 300M blog articles. The records are in html format. Is it possible to import these records using only Solr CELL+TIKA+DIH to some Solr with schema? I mean when importing, I can map schema on mysql to schema in Solr?
scott.chu，scott.chu@udngroup.com
2016/5/20 (週五)
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12262 - 發佈日期: 05/19/16
Re: Import html data in mysql and map schemas using
onlySolrCELL+TIKA+DIH [scottchu]
Posted by Tom Evans <te...@googlemail.com>.
On Tue, May 24, 2016 at 3:06 PM, Scott Chu <sc...@udngroup.com> wrote:
> p.s. There're really many many extensive, worthy stuffs in Solr. If the
> project team can provide some "dictionary" of them, It would be a "Santa Claus"
> for we solr users. Ha! Just a X'mas wish! Sigh! I know it's quite not possbile.
> I really like to study them one after another, to learn about all of them.
> However, Internet IT goes too fast to have time to congest all of the great
> stuffs in Solr.
The reference guide is both extensive and also broadly informative.
Start from the top page and browse away!
https://cwiki.apache.org/confluence/display/solr/Apache+Solr+Reference+Guide
Handy to keep the glossary handy for any terms that you don't recognise:
https://cwiki.apache.org/confluence/display/solr/Solr+Glossary
Cheers
Tom
Re: Import html data in mysql and map schemas using
onlySolrCELL+TIKA+DIH [scottchu]
Posted by Scott Chu <sc...@udngroup.com>.
Justa let everybody know. I use DIH+template (without TIKA and Solr Cell, I really don't understand that part in reference guide) to achieve what I want. But still need to test more various form of HTML source.
Scott Chu，scott.chu@udngroup.com
2016/5/24 (週二)
p.s. There're really many many extensive, worthy stuffs in Solr. If the project team can provide some "dictionary" of them, It would be a "Santa Claus" for we solr users. Ha! Just a X'mas wish! Sigh! I know it's quite not possbile. I really like to study them one after another, to learn about all of them. However, Internet IT goes too fast to have time to congest all of the great stuffs in Solr.
----- Original Message -----
From: scott.chu
To: solr-user
CC:
Date: 2016/5/21 (週六) 03:39
Subject: Re: Import html data in mysql and map schemas using onlySolrCELL+TIKA+DIH [scottchu]
For this project, I intend to use Solr 5.5 or Solr 6. I know how to modify config to go back to use ClassicIndex, ie. manual schema.xml.
Scott Chu，scott.chu@udngroup.com
2016/5/21 (週六)
----- Original Message -----
From: Siddhartha Singh Sandhu
To: solr-user ; scott.chu
CC:
Date: 2016/5/21 (週六) 03:33
Subject: Re: Import html data in mysql and map schemas using only SolrCELL+TIKA+DIH [scottchu]
You will have to configure your schema.xml in Solr.
What version are you using?
On Fri, May 20, 2016 at 2:17 AM, scott.chu <sc...@udngroup.com> wrote:
>
> I have a mysql table with over 300M blog articles. The records are in html
> format. Is it possible to import these records using only Solr
> CELL+TIKA+DIH to some Solr with schema? I mean when importing, I can map
> schema on mysql to schema in Solr?
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/20 (週五)
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12265 - 發佈日期: 05/20/16
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12265 - 發佈日期: 05/20/16
Re: Import html data in mysql and map schemas using only
SolrCELL+TIKA+DIH [scottchu]
Posted by Scott Chu <sc...@udngroup.com>.
For this project, I intend to use Solr 5.5 or Solr 6. I know how to modify config to go back to use ClassicIndex, ie. manual schema.xml.
Scott Chu，scott.chu@udngroup.com
2016/5/21 (週六)
----- Original Message -----
From: Siddhartha Singh Sandhu
To: solr-user ; scott.chu
CC:
Date: 2016/5/21 (週六) 03:33
Subject: Re: Import html data in mysql and map schemas using only SolrCELL+TIKA+DIH [scottchu]
You will have to configure your schema.xml in Solr.
What version are you using?
On Fri, May 20, 2016 at 2:17 AM, scott.chu <sc...@udngroup.com> wrote:
>
> I have a mysql table with over 300M blog articles. The records are in html
> format. Is it possible to import these records using only Solr
> CELL+TIKA+DIH to some Solr with schema? I mean when importing, I can map
> schema on mysql to schema in Solr?
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/20 (週五)
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12265 - 發佈日期: 05/20/16
Re: Import html data in mysql and map schemas using only Solr
CELL+TIKA+DIH [scottchu]
Posted by Siddhartha Singh Sandhu <sa...@gmail.com>.
You will have to configure your schema.xml in Solr.
What version are you using?
On Fri, May 20, 2016 at 2:17 AM, scott.chu <sc...@udngroup.com> wrote:
>
> I have a mysql table with over 300M blog articles. The records are in html
> format. Is it possible to import these records using only Solr
> CELL+TIKA+DIH to some Solr with schema? I mean when importing, I can map
> schema on mysql to schema in Solr?
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/20 (週五)
>
Import html data in mysql and map schemas using only Solr
CELL+TIKA+DIH [scottchu]
Posted by "scott.chu" <sc...@udngroup.com>.
I have a mysql table with over 300M blog articles. The records are in html format. Is it possible to import these records using only Solr CELL+TIKA+DIH to some Solr with schema? I mean when importing, I can map schema on mysql to schema in Solr?
scott.chu，scott.chu@udngroup.com
2016/5/20 (週五)
Re: [scottchu] How to specify multiple zk nodes using
solrstartcommand under Windows
Posted by "scott.chu" <sc...@udngroup.com>.
I find a way to stop zk nodes peacefully under Windows.
If you start zk nodes in order of 1,2,3
then you should stop them in reverse order, i.e. 3,2,1
Thus you can stop process peacefully without any errors.
Hope this help people!
scott.chu，scott.chu@udngroup.com
2016/5/19 (週四)
----- Original Message -----
From: John Bickerstaff
To: solr-user ; scott(自己)
CC:
Date: 2016/5/18 (週三) 13:53
Subject: Re: [scottchu] How to specify multiple zk nodes using solrstartcommand under Windows
I think those zk server warning messages are expected. Until you have 3
running instances you don't have a "Quorum" and the Zookeeper instances
complain. Once the third one comes up they are "happy" and don't complain
any more. You'd get similar messages if one of the Zookeeper nodes ever
went down.
As for the stopping of zk server - I've never had any problem issuing a
stop command, but I'm running Linux so I may not be much good to you in
that regard.
On Tue, May 17, 2016 at 8:41 PM, scott.chu <sc...@udngroup.com> wrote:
> I tested yesterday and it proves my theory. I'll share what I do under
> Windows on 1 PC here with you experienced guys and further newbies:
>
> 1>Download zookeeper 3.4.8. I unzip it and copy to 3 other different
> folders: zk_1, zk_2, zk_3.
> 2>For each zk_n folder, I do these things (Note: {n} means the last digit
> in zk_n foler name):
> a. Create zoo_data folder under root and create 'myid' with
> notepad, the contents is pure '{n}' only.
> b. Create zoo.cfg under conf folder with following contents:
> clientPort=218{n}
> initLimit=5
> syncLimit=2
> dataDir=D:/zk_{n}/zoo_data
> ;if p2p-coneect-port or leader-election-port are all
> same, then we should set maxClientCnxns=n
> ;maxClientCnxns=3
> ;server.x=host:p2p-connect-port:leader-election-port
> server.1=localhost:2888:3888
> server.2=localhost:2889:3889
> server.3=localhost:2890:3890
> 3> I download ZOOKEEPER-1122's zkServer.cmd. and go into each zk_n folder
> and issue command:
> bin\zkServer.cmd start
>
> [Question]: There's something I'd like to ask guys: When I start
> zk_1, zk_2, the console keeps shows some warning messages.
> Only after I start zk_3, the warning messages
> is stopped. Is that normal?
>
> 4> I use zkui_win to see them all go online successfully.
> 5> I goto Solr-5.4.1 folder, and issue following commands:
> bin\solr start -c -s mynodes\node1 -z localhost:2181
> bin\solr start -c -s mynodes\node1 -z localhost:2181 -p
> 7973
> bin\solr create -c cugna -d myconfigsets\cugna -shards
> 1 -replicationFactor 2 -p 8983
> 6> By using zkui_win again, I see:
> ****** config 'cugna' are all synchronized on zk_1 to zk_3. So this
> proves my theory, we only have to specify only one zk nodes and they'll
> sync themselves. ******
>
> [Question]: I go into zk_n folder and issue 'bin\zkServer stop'. However,
> this shows error message. It seems it can't taskkill the zk process for
> some reason. The only way I stop them
> is by closing DOS windows that has issued the
> 'bin\zkServer start' command. Does anybody know why 'bin\zkServer stop'
> doesn't work?
>
> Note: Gotta say sorry for the repitition of localhost:2181. It's my typo.
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/18 (週三)
> ----- Original Message -----
> From: Abdel Belkasri
> To: solr-user
> CC:
> Date: 2016/5/18 (週三) 00:17
> Subject: Re: [scottchu] How to specify multiple zk nodes using solr
> startcommand under Windows
>
>
> The repetition is just a cut and paste from Scott's post.
>
> How can I check if I am getting the ensemble or just a single zk?
>
> Also if this is not the way to specify an ensemble, what is the right way?
>
>
> Because the comma delimited list does not work, I concur with Scott.
>
> On Tue, May 17, 2016 at 11:49 AM, Erick Erickson <er...@gmail.com>
>
> wrote:
>
> > Are you absolutely sure you're getting an _ensemble_ and
> > not just connecting to a single node? My suspicion (without
> > proof) is that you're just getting one -z option. It'll work as
> > long as that ZK instance stays up, but it won't be fault-tolerant.
> >
> > And again you repeated the port (2181) twice.
> >
> > Best,
> > Erick
> >
> > On Tue, May 17, 2016 at 8:02 AM, Abdel Belkasri <be...@gmail.com>
> > wrote:
> > > Hi Scott,
> > > what worked for me in Windows is this (no ",")
> > > bin\Solr start -c -s mynodes\node1 -z localhost:2181 -z localhost:2181
> -z
> > > localhost:2183
> > >
> > > -- Hope this helps
> > > Abdel.
> > >
> > > On Tue, May 17, 2016 at 3:35 AM, scott.chu <sc...@udngroup.com>
> > wrote:
> > >
> > >> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
> > >> Go into Solr 5.4.1 root folder and issue and issue the command in
> > article
> > >> 'Setting Up an External ZooKeeper Ensemble' in reference guide
> > >>
> > >> bin\Solr start -c -s mynodes\node1 -z
> > >> localhost:2181,localhost:2181,localhost:2183
> > >>
> > >> but it doesn't run but just show help page of start command in
> solr.cmd.
> > >> How should I issue the correct command?
> > >>
> > >
> > >
> > >
> > > --
> > > Abdel K. Belkasri, PhD
> >
>
>
>
> --
> Abdel K. Belkasri, PhD
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12251 - 發佈日期: 05/17/16
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12251 - 發佈日期: 05/17/16
Re: [scottchu] How to specify multiple zk nodes using
solrstartcommand under Windows
Posted by "scott.chu" <sc...@udngroup.com>.
Thanks! Once I do experiment ok under Windows. I'll move to CentOS. Actually, I already install 3 zk and Solr on 3 servers with CentOS. It seems that open source project more or less has something unprepared for running under Windows.
scott.chu，scott.chu@udngroup.com
2016/5/18 (週三)
----- Original Message -----
From: John Bickerstaff
To: solr-user ; scott(自己)
CC:
Date: 2016/5/18 (週三) 13:53
Subject: Re: [scottchu] How to specify multiple zk nodes using solrstartcommand under Windows
I think those zk server warning messages are expected. Until you have 3
running instances you don't have a "Quorum" and the Zookeeper instances
complain. Once the third one comes up they are "happy" and don't complain
any more. You'd get similar messages if one of the Zookeeper nodes ever
went down.
As for the stopping of zk server - I've never had any problem issuing a
stop command, but I'm running Linux so I may not be much good to you in
that regard.
On Tue, May 17, 2016 at 8:41 PM, scott.chu <sc...@udngroup.com> wrote:
> I tested yesterday and it proves my theory. I'll share what I do under
> Windows on 1 PC here with you experienced guys and further newbies:
>
> 1>Download zookeeper 3.4.8. I unzip it and copy to 3 other different
> folders: zk_1, zk_2, zk_3.
> 2>For each zk_n folder, I do these things (Note: {n} means the last digit
> in zk_n foler name):
> a. Create zoo_data folder under root and create 'myid' with
> notepad, the contents is pure '{n}' only.
> b. Create zoo.cfg under conf folder with following contents:
> clientPort=218{n}
> initLimit=5
> syncLimit=2
> dataDir=D:/zk_{n}/zoo_data
> ;if p2p-coneect-port or leader-election-port are all
> same, then we should set maxClientCnxns=n
> ;maxClientCnxns=3
> ;server.x=host:p2p-connect-port:leader-election-port
> server.1=localhost:2888:3888
> server.2=localhost:2889:3889
> server.3=localhost:2890:3890
> 3> I download ZOOKEEPER-1122's zkServer.cmd. and go into each zk_n folder
> and issue command:
> bin\zkServer.cmd start
>
> [Question]: There's something I'd like to ask guys: When I start
> zk_1, zk_2, the console keeps shows some warning messages.
> Only after I start zk_3, the warning messages
> is stopped. Is that normal?
>
> 4> I use zkui_win to see them all go online successfully.
> 5> I goto Solr-5.4.1 folder, and issue following commands:
> bin\solr start -c -s mynodes\node1 -z localhost:2181
> bin\solr start -c -s mynodes\node1 -z localhost:2181 -p
> 7973
> bin\solr create -c cugna -d myconfigsets\cugna -shards
> 1 -replicationFactor 2 -p 8983
> 6> By using zkui_win again, I see:
> ****** config 'cugna' are all synchronized on zk_1 to zk_3. So this
> proves my theory, we only have to specify only one zk nodes and they'll
> sync themselves. ******
>
> [Question]: I go into zk_n folder and issue 'bin\zkServer stop'. However,
> this shows error message. It seems it can't taskkill the zk process for
> some reason. The only way I stop them
> is by closing DOS windows that has issued the
> 'bin\zkServer start' command. Does anybody know why 'bin\zkServer stop'
> doesn't work?
>
> Note: Gotta say sorry for the repitition of localhost:2181. It's my typo.
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/18 (週三)
> ----- Original Message -----
> From: Abdel Belkasri
> To: solr-user
> CC:
> Date: 2016/5/18 (週三) 00:17
> Subject: Re: [scottchu] How to specify multiple zk nodes using solr
> startcommand under Windows
>
>
> The repetition is just a cut and paste from Scott's post.
>
> How can I check if I am getting the ensemble or just a single zk?
>
> Also if this is not the way to specify an ensemble, what is the right way?
>
>
> Because the comma delimited list does not work, I concur with Scott.
>
> On Tue, May 17, 2016 at 11:49 AM, Erick Erickson <er...@gmail.com>
>
> wrote:
>
> > Are you absolutely sure you're getting an _ensemble_ and
> > not just connecting to a single node? My suspicion (without
> > proof) is that you're just getting one -z option. It'll work as
> > long as that ZK instance stays up, but it won't be fault-tolerant.
> >
> > And again you repeated the port (2181) twice.
> >
> > Best,
> > Erick
> >
> > On Tue, May 17, 2016 at 8:02 AM, Abdel Belkasri <be...@gmail.com>
> > wrote:
> > > Hi Scott,
> > > what worked for me in Windows is this (no ",")
> > > bin\Solr start -c -s mynodes\node1 -z localhost:2181 -z localhost:2181
> -z
> > > localhost:2183
> > >
> > > -- Hope this helps
> > > Abdel.
> > >
> > > On Tue, May 17, 2016 at 3:35 AM, scott.chu <sc...@udngroup.com>
> > wrote:
> > >
> > >> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
> > >> Go into Solr 5.4.1 root folder and issue and issue the command in
> > article
> > >> 'Setting Up an External ZooKeeper Ensemble' in reference guide
> > >>
> > >> bin\Solr start -c -s mynodes\node1 -z
> > >> localhost:2181,localhost:2181,localhost:2183
> > >>
> > >> but it doesn't run but just show help page of start command in
> solr.cmd.
> > >> How should I issue the correct command?
> > >>
> > >
> > >
> > >
> > > --
> > > Abdel K. Belkasri, PhD
> >
>
>
>
> --
> Abdel K. Belkasri, PhD
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12251 - 發佈日期: 05/17/16
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12251 - 發佈日期: 05/17/16
Re: [scottchu] How to specify multiple zk nodes using solr
startcommand under Windows
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
I think those zk server warning messages are expected. Until you have 3
running instances you don't have a "Quorum" and the Zookeeper instances
complain. Once the third one comes up they are "happy" and don't complain
any more. You'd get similar messages if one of the Zookeeper nodes ever
went down.
As for the stopping of zk server - I've never had any problem issuing a
stop command, but I'm running Linux so I may not be much good to you in
that regard.
On Tue, May 17, 2016 at 8:41 PM, scott.chu <sc...@udngroup.com> wrote:
> I tested yesterday and it proves my theory. I'll share what I do under
> Windows on 1 PC here with you experienced guys and further newbies:
>
> 1>Download zookeeper 3.4.8. I unzip it and copy to 3 other different
> folders: zk_1, zk_2, zk_3.
> 2>For each zk_n folder, I do these things (Note: {n} means the last digit
> in zk_n foler name):
> a. Create zoo_data folder under root and create 'myid' with
> notepad, the contents is pure '{n}' only.
> b. Create zoo.cfg under conf folder with following contents:
> clientPort=218{n}
> initLimit=5
> syncLimit=2
> dataDir=D:/zk_{n}/zoo_data
> ;if p2p-coneect-port or leader-election-port are all
> same, then we should set maxClientCnxns=n
> ;maxClientCnxns=3
> ;server.x=host:p2p-connect-port:leader-election-port
> server.1=localhost:2888:3888
> server.2=localhost:2889:3889
> server.3=localhost:2890:3890
> 3> I download ZOOKEEPER-1122's zkServer.cmd. and go into each zk_n folder
> and issue command:
> bin\zkServer.cmd start
>
> [Question]: There's something I'd like to ask guys: When I start
> zk_1, zk_2, the console keeps shows some warning messages.
> Only after I start zk_3, the warning messages
> is stopped. Is that normal?
>
> 4> I use zkui_win to see them all go online successfully.
> 5> I goto Solr-5.4.1 folder, and issue following commands:
> bin\solr start -c -s mynodes\node1 -z localhost:2181
> bin\solr start -c -s mynodes\node1 -z localhost:2181 -p
> 7973
> bin\solr create -c cugna -d myconfigsets\cugna -shards
> 1 -replicationFactor 2 -p 8983
> 6> By using zkui_win again, I see:
> ****** config 'cugna' are all synchronized on zk_1 to zk_3. So this
> proves my theory, we only have to specify only one zk nodes and they'll
> sync themselves. ******
>
> [Question]: I go into zk_n folder and issue 'bin\zkServer stop'. However,
> this shows error message. It seems it can't taskkill the zk process for
> some reason. The only way I stop them
> is by closing DOS windows that has issued the
> 'bin\zkServer start' command. Does anybody know why 'bin\zkServer stop'
> doesn't work?
>
> Note: Gotta say sorry for the repitition of localhost:2181. It's my typo.
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/18 (週三)
> ----- Original Message -----
> From: Abdel Belkasri
> To: solr-user
> CC:
> Date: 2016/5/18 (週三) 00:17
> Subject: Re: [scottchu] How to specify multiple zk nodes using solr
> startcommand under Windows
>
>
> The repetition is just a cut and paste from Scott's post.
>
> How can I check if I am getting the ensemble or just a single zk?
>
> Also if this is not the way to specify an ensemble, what is the right way?
>
>
> Because the comma delimited list does not work, I concur with Scott.
>
> On Tue, May 17, 2016 at 11:49 AM, Erick Erickson <er...@gmail.com>
>
> wrote:
>
> > Are you absolutely sure you're getting an _ensemble_ and
> > not just connecting to a single node? My suspicion (without
> > proof) is that you're just getting one -z option. It'll work as
> > long as that ZK instance stays up, but it won't be fault-tolerant.
> >
> > And again you repeated the port (2181) twice.
> >
> > Best,
> > Erick
> >
> > On Tue, May 17, 2016 at 8:02 AM, Abdel Belkasri <be...@gmail.com>
> > wrote:
> > > Hi Scott,
> > > what worked for me in Windows is this (no ",")
> > > bin\Solr start -c -s mynodes\node1 -z localhost:2181 -z localhost:2181
> -z
> > > localhost:2183
> > >
> > > -- Hope this helps
> > > Abdel.
> > >
> > > On Tue, May 17, 2016 at 3:35 AM, scott.chu <sc...@udngroup.com>
> > wrote:
> > >
> > >> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
> > >> Go into Solr 5.4.1 root folder and issue and issue the command in
> > article
> > >> 'Setting Up an External ZooKeeper Ensemble' in reference guide
> > >>
> > >> bin\Solr start -c -s mynodes\node1 -z
> > >> localhost:2181,localhost:2181,localhost:2183
> > >>
> > >> but it doesn't run but just show help page of start command in
> solr.cmd.
> > >> How should I issue the correct command?
> > >>
> > >
> > >
> > >
> > > --
> > > Abdel K. Belkasri, PhD
> >
>
>
>
> --
> Abdel K. Belkasri, PhD
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12251 - 發佈日期: 05/17/16
>
A quick way to open stay-opened DOS window on any folder under
Windows [scottchu]
Posted by "scott.chu" <sc...@udngroup.com>.
I got many helps from this maillist. So this time I'd like to share some useful thing with guys.
First, I'm not a cygwin guy under Windows.
[Symptom]
When we test some open source projects under Windows. We from time to time need: open Windows explorer, change to specific folder, click once on top address bar, select and copy the path, open DOS window, issue 'cd /d ', right-click to paste the pathname, press ENTER, and then we can type in some commands, etc. This could be tedious if we test this, test that. At least it 's what I'm experienced these days.
[Cure]
I find a quick and universal way to do previous tedious actions, as follows:
1> Create a cmd.exe link.
2> Rename it to 'cdhere'.
3> Right-click on it and select 'properties'.
4>On textbox of target, type in 'C:\Windows\System\cmd.exe'.
5>On textbox of starting position, type in '%cd%'.
6>Press 'OK'.
Now you can copy cdhere.lnk to any folder you want, double-click it, and the DOS windows will be opened and stayed.
If you are kinda like me (i.e. not a cygwin guy), then this should help you. Enjoy it!
scott.chu，scott.chu@udngroup.com
2016/5/18 (週三 means Wednesday)
Re: [scottchu] How to specify multiple zk nodes using solr
startcommand under Windows
Posted by "scott.chu" <sc...@udngroup.com>.
I tested yesterday and it proves my theory. I'll share what I do under Windows on 1 PC here with you experienced guys and further newbies:
1>Download zookeeper 3.4.8. I unzip it and copy to 3 other different folders: zk_1, zk_2, zk_3.
2>For each zk_n folder, I do these things (Note: {n} means the last digit in zk_n foler name):
a. Create zoo_data folder under root and create 'myid' with notepad, the contents is pure '{n}' only.
b. Create zoo.cfg under conf folder with following contents:
clientPort=218{n}
initLimit=5
syncLimit=2
dataDir=D:/zk_{n}/zoo_data
;if p2p-coneect-port or leader-election-port are all same, then we should set maxClientCnxns=n
;maxClientCnxns=3
;server.x=host:p2p-connect-port:leader-election-port
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
3> I download ZOOKEEPER-1122's zkServer.cmd. and go into each zk_n folder and issue command:
bin\zkServer.cmd start
[Question]: There's something I'd like to ask guys: When I start zk_1, zk_2, the console keeps shows some warning messages.
Only after I start zk_3, the warning messages is stopped. Is that normal?
4> I use zkui_win to see them all go online successfully.
5> I goto Solr-5.4.1 folder, and issue following commands:
bin\solr start -c -s mynodes\node1 -z localhost:2181
bin\solr start -c -s mynodes\node1 -z localhost:2181 -p 7973
bin\solr create -c cugna -d myconfigsets\cugna -shards 1 -replicationFactor 2 -p 8983
6> By using zkui_win again, I see:
****** config 'cugna' are all synchronized on zk_1 to zk_3. So this proves my theory, we only have to specify only one zk nodes and they'll sync themselves. ******
[Question]: I go into zk_n folder and issue 'bin\zkServer stop'. However, this shows error message. It seems it can't taskkill the zk process for some reason. The only way I stop them
is by closing DOS windows that has issued the 'bin\zkServer start' command. Does anybody know why 'bin\zkServer stop' doesn't work?
Note: Gotta say sorry for the repitition of localhost:2181. It's my typo.
scott.chu，scott.chu@udngroup.com
2016/5/18 (週三)
----- Original Message -----
From: Abdel Belkasri
To: solr-user
CC:
Date: 2016/5/18 (週三) 00:17
Subject: Re: [scottchu] How to specify multiple zk nodes using solr startcommand under Windows
The repetition is just a cut and paste from Scott's post.
How can I check if I am getting the ensemble or just a single zk?
Also if this is not the way to specify an ensemble, what is the right way?
Because the comma delimited list does not work, I concur with Scott.
On Tue, May 17, 2016 at 11:49 AM, Erick Erickson <er...@gmail.com>
wrote:
> Are you absolutely sure you're getting an _ensemble_ and
> not just connecting to a single node? My suspicion (without
> proof) is that you're just getting one -z option. It'll work as
> long as that ZK instance stays up, but it won't be fault-tolerant.
>
> And again you repeated the port (2181) twice.
>
> Best,
> Erick
>
> On Tue, May 17, 2016 at 8:02 AM, Abdel Belkasri <be...@gmail.com>
> wrote:
> > Hi Scott,
> > what worked for me in Windows is this (no ",")
> > bin\Solr start -c -s mynodes\node1 -z localhost:2181 -z localhost:2181 -z
> > localhost:2183
> >
> > -- Hope this helps
> > Abdel.
> >
> > On Tue, May 17, 2016 at 3:35 AM, scott.chu <sc...@udngroup.com>
> wrote:
> >
> >> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
> >> Go into Solr 5.4.1 root folder and issue and issue the command in
> article
> >> 'Setting Up an External ZooKeeper Ensemble' in reference guide
> >>
> >> bin\Solr start -c -s mynodes\node1 -z
> >> localhost:2181,localhost:2181,localhost:2183
> >>
> >> but it doesn't run but just show help page of start command in solr.cmd.
> >> How should I issue the correct command?
> >>
> >
> >
> >
> > --
> > Abdel K. Belkasri, PhD
>
--
Abdel K. Belkasri, PhD
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12251 - 發佈日期: 05/17/16
Re: [scottchu] How to specify multiple zk nodes using solr start
command under Windows
Posted by Abdel Belkasri <be...@gmail.com>.
The repetition is just a cut and paste from Scott's post.
How can I check if I am getting the ensemble or just a single zk?
Also if this is not the way to specify an ensemble, what is the right way?
Because the comma delimited list does not work, I concur with Scott.
On Tue, May 17, 2016 at 11:49 AM, Erick Erickson <er...@gmail.com>
wrote:
> Are you absolutely sure you're getting an _ensemble_ and
> not just connecting to a single node? My suspicion (without
> proof) is that you're just getting one -z option. It'll work as
> long as that ZK instance stays up, but it won't be fault-tolerant.
>
> And again you repeated the port (2181) twice.
>
> Best,
> Erick
>
> On Tue, May 17, 2016 at 8:02 AM, Abdel Belkasri <be...@gmail.com>
> wrote:
> > Hi Scott,
> > what worked for me in Windows is this (no ",")
> > bin\Solr start -c -s mynodes\node1 -z localhost:2181 -z localhost:2181 -z
> > localhost:2183
> >
> > -- Hope this helps
> > Abdel.
> >
> > On Tue, May 17, 2016 at 3:35 AM, scott.chu <sc...@udngroup.com>
> wrote:
> >
> >> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
> >> Go into Solr 5.4.1 root folder and issue and issue the command in
> article
> >> 'Setting Up an External ZooKeeper Ensemble' in reference guide
> >>
> >> bin\Solr start -c -s mynodes\node1 -z
> >> localhost:2181,localhost:2181,localhost:2183
> >>
> >> but it doesn't run but just show help page of start command in solr.cmd.
> >> How should I issue the correct command?
> >>
> >
> >
> >
> > --
> > Abdel K. Belkasri, PhD
>
--
Abdel K. Belkasri, PhD
Re: [scottchu] How to specify multiple zk nodes using solr start
command under Windows
Posted by Erick Erickson <er...@gmail.com>.
Are you absolutely sure you're getting an _ensemble_ and
not just connecting to a single node? My suspicion (without
proof) is that you're just getting one -z option. It'll work as
long as that ZK instance stays up, but it won't be fault-tolerant.
And again you repeated the port (2181) twice.
Best,
Erick
On Tue, May 17, 2016 at 8:02 AM, Abdel Belkasri <be...@gmail.com> wrote:
> Hi Scott,
> what worked for me in Windows is this (no ",")
> bin\Solr start -c -s mynodes\node1 -z localhost:2181 -z localhost:2181 -z
> localhost:2183
>
> -- Hope this helps
> Abdel.
>
> On Tue, May 17, 2016 at 3:35 AM, scott.chu <sc...@udngroup.com> wrote:
>
>> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
>> Go into Solr 5.4.1 root folder and issue and issue the command in article
>> 'Setting Up an External ZooKeeper Ensemble' in reference guide
>>
>> bin\Solr start -c -s mynodes\node1 -z
>> localhost:2181,localhost:2181,localhost:2183
>>
>> but it doesn't run but just show help page of start command in solr.cmd.
>> How should I issue the correct command?
>>
>
>
>
> --
> Abdel K. Belkasri, PhD
Re: [scottchu] How to specify multiple zk nodes using solr start
command under Windows
Posted by Abdel Belkasri <be...@gmail.com>.
Hi Scott,
what worked for me in Windows is this (no ",")
bin\Solr start -c -s mynodes\node1 -z localhost:2181 -z localhost:2181 -z
localhost:2183
-- Hope this helps
Abdel.
On Tue, May 17, 2016 at 3:35 AM, scott.chu <sc...@udngroup.com> wrote:
> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
> Go into Solr 5.4.1 root folder and issue and issue the command in article
> 'Setting Up an External ZooKeeper Ensemble' in reference guide
>
> bin\Solr start -c -s mynodes\node1 -z
> localhost:2181,localhost:2181,localhost:2183
>
> but it doesn't run but just show help page of start command in solr.cmd.
> How should I issue the correct command?
>
--
Abdel K. Belkasri, PhD
Re: [scottchu] How to specify multiple zk nodes using solr start
commandunder Windows
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
In your original command, you listed the same port twice. That may have
been at least part of the difficulty.
It's probably fine to just use one zk node - as the zookeeper instances
should be aware of each other.
I also assume that if your solr.in.sh (or windows equavalent) has the
properly formatted entry for all the zk nodes, Solr will be able to find a
different one if the one you pass in goes down...
Here's the entry from my file in case it's helpful. I got away without the
port number (I assume) because I'm using the default 2181 on all my
zookeeper nodes which are separate servers.
# Set the ZooKeeper connection string if using an external ZooKeeper
ensemble
# e.g. host1:2181,host2:2181/chroot
# Leave empty if not using SolrCloud
ZK_HOST="192.168.56.5,192.168.56.6,192.168.56.7/solr5_4"
On Tue, May 17, 2016 at 4:30 AM, scott.chu <sc...@udngroup.com> wrote:
>
> I issue '-z localhost:2181 -z localhost:2182 -z localhost:2183' for each
> node's start command and later when I create collection, all 3 zk nodes has
> registered my configset.
> Never try but I think maybe only use -z localhost:2181, then all 3 nodes
> in zk ensemble will synchronize themselves.
>
> scott.chu，scott.chu@udngroup.com
> 2016/5/17 (週二)
> ----- Original Message -----
> From: scott(自己)
> To: solr-user
> CC:
> Date: 2016/5/17 (週二) 15:35
> Subject: [scottchu] How to specify multiple zk nodes using solr start
> commandunder Windows
>
>
> I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
> Go into Solr 5.4.1 root folder and issue and issue the command in article
> 'Setting Up an External ZooKeeper Ensemble' in reference guide
>
> bin\Solr start -c -s mynodes\node1 -z
> localhost:2181,localhost:2181,localhost:2183
>
> but it doesn't run but just show help page of start command in solr.cmd.
> How should I issue the correct command?
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12245 - 發佈日期: 05/16/16
>
Re: [scottchu] How to specify multiple zk nodes using solr start
commandunder Windows
Posted by "scott.chu" <sc...@udngroup.com>.
I issue '-z localhost:2181 -z localhost:2182 -z localhost:2183' for each node's start command and later when I create collection, all 3 zk nodes has registered my configset.
Never try but I think maybe only use -z localhost:2181, then all 3 nodes in zk ensemble will synchronize themselves.
scott.chu，scott.chu@udngroup.com
2016/5/17 (週二)
----- Original Message -----
From: scott(自己)
To: solr-user
CC:
Date: 2016/5/17 (週二) 15:35
Subject: [scottchu] How to specify multiple zk nodes using solr start commandunder Windows
I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
Go into Solr 5.4.1 root folder and issue and issue the command in article 'Setting Up an External ZooKeeper Ensemble' in reference guide
bin\Solr start -c -s mynodes\node1 -z localhost:2181,localhost:2181,localhost:2183
but it doesn't run but just show help page of start command in solr.cmd. How should I issue the correct command?
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12245 - 發佈日期: 05/16/16
[scottchu] How to specify multiple zk nodes using solr start command
under Windows
Posted by "scott.chu" <sc...@udngroup.com>.
I start 3 zk nodes at port 2181,2182, and 2183 on my local machine.
Go into Solr 5.4.1 root folder and issue and issue the command in article 'Setting Up an External ZooKeeper Ensemble' in reference guide
bin\Solr start -c -s mynodes\node1 -z localhost:2181,localhost:2181,localhost:2183
but it doesn't run but just show help page of start command in solr.cmd. How should I issue the correct command?
Re: Updating error while add doc to Solrcloud
Posted by "scott.chu" <sc...@udngroup.com>.
I clear the cugna collection data (by renaming 'data' folder to 'xdata')and restart Solrcloud. I add previous possible-error xml doc, it succeeds. So I'm sure doc data has no problem. Is it because the index file size is too large? If the zk nodes fails during adding doc, could it cause this updating error since I do see some "can't find leader error' in solr log?
scott.chu，scott.chu@udngroup.com
2016/5/17 (週二)
----- Original Message -----
From: scott(自己)
To: solr-user
CC:
Date: 2016/5/17 (週二) 14:29
Subject: Updating error while add doc to Solrcloud
I build Solrcloud with 2 nodes, 1 shard, 2 replica. I add doc in xml format using post.jar up to 2.85M+ no. of docs and 10gb index size. When I add more docs. the solr.log shows:
--------------------------------------
2016-05-17 14:01:09,024 WARN (main) [ ] o.e.j.s.h.RequestLogHandler !RequestLog
2016-05-17 14:01:09,275 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@57fffcd7{/solr,file:/D:/portable_sw/solr-5.4.1/server/solr-webapp/webapp/,STARTING}{D:\portable_sw\solr-5.4.1\server/solr-webapp/webapp} has uncovered http methods for path: /
2016-05-17 14:01:09,346 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from D:\portable_sw\solr-5.4.1\mynodes\cloud\node1\lib to classpath: D:\portable_sw\solr-5.4.1\mynodes\cloud\node1\lib
2016-05-17 14:01:11,419 WARN (coreLoadExecutor-7-thread-1-processing-n:10.18.59.179:8983_solr) [c:cugna s:shard1 r:core_node2 x:cugna_shard1_replica2] o.a.s.u.UpdateLog Exception reverse reading log
java.io.EOFException
...
--------------------------------------
Later I stop all and kill write.lock (I ususally do this in Solr 3 when add doc fails) and add doc again but Solrcloud show can't find write.lock. So I recover write.lock and call post.jar again. The output shows:
--------------------------------------
"SimplePostTool version 5.0.0
Posting files to [base] url http://localhost:8983/solr/cugna/update using content-type application/xml...
POSTing file NMLBOym_a_UN2000_10_20160511_1014.xml to [base]
SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url: http://localhost:8983/solr/cugna/update
SimplePostTool: WARNING: Response: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">400</int><int name="QTime">3</int></lst><lst name="error"><str name="msg">
Exception writing document id un_555917 to the index; possible analysis error.</str><int name="code">400</int></lst>
</response>
SimplePostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 40
0 for URL: http://localhost:8983/solr/cugna/update
1 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/cugna/update...
Time spent: 0:00:00.259"
-------------------------------------
I thought it's that doc un_555917 problem, then I comment out it in xml and do again, it keeps showing same error to every single doc. I assume there's something wrong with Solrcloud. Does anyone experience this before? What could be the problem? Can I recover it? Or I have to add all docs again?
scott.chu，scott.chu@udngroup.com
2016/5/17 (週二)
----- Original Message -----
From: scott(自己)
To: solr-user
CC:
Date: 2016/5/17 (週二) 09:39
Subject: Re(2): [scottchu] Cab I migrate solrcloud by just copying wholepackagefolder?
OK! Thanks for reminding. I'll stick to the convention.
scott.chu，scott.chu@udngroup.com
2016/5/17 (週二)
----- Original Message -----
From: Chris Hostetter
To: solr-user ; scott(自己)
CC:
Date: 2016/5/17 (週二) 02:43
Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole packagefolder?
: Message-Id: <7F...@udngroup.com>
: Subject: [scottchu] Cab I migrate solrcloud by just copying whole package
: folder?
: References:
: <DC...@udngroup.com><C162E1B1F0311FC46A64E2E2AA
: 729138@udngroup.com>,
: <85...@elyograg.org>
: In-Reply-To: <85...@elyograg.org>
https://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
-Hoss
http://www.lucidworks.com/
-----
???????????
??? AVG ?? - www.avg.com
??: 2015.0.6201 / ???: 4568/12238 - ????: 05/15/16
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12245 - 發佈日期: 05/16/16
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12245 - 發佈日期: 05/16/16
Updating error while add doc to Solrcloud
Posted by "scott.chu" <sc...@udngroup.com>.
I build Solrcloud with 2 nodes, 1 shard, 2 replica. I add doc in xml format using post.jar up to 2.85M+ no. of docs and 10gb index size. When I add more docs. the solr.log shows:
--------------------------------------
2016-05-17 14:01:09,024 WARN (main) [ ] o.e.j.s.h.RequestLogHandler !RequestLog
2016-05-17 14:01:09,275 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@57fffcd7{/solr,file:/D:/portable_sw/solr-5.4.1/server/solr-webapp/webapp/,STARTING}{D:\portable_sw\solr-5.4.1\server/solr-webapp/webapp} has uncovered http methods for path: /
2016-05-17 14:01:09,346 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from D:\portable_sw\solr-5.4.1\mynodes\cloud\node1\lib to classpath: D:\portable_sw\solr-5.4.1\mynodes\cloud\node1\lib
2016-05-17 14:01:11,419 WARN (coreLoadExecutor-7-thread-1-processing-n:10.18.59.179:8983_solr) [c:cugna s:shard1 r:core_node2 x:cugna_shard1_replica2] o.a.s.u.UpdateLog Exception reverse reading log
java.io.EOFException
...
--------------------------------------
Later I stop all and kill write.lock (I ususally do this in Solr 3 when add doc fails) and add doc again but Solrcloud show can't find write.lock. So I recover write.lock and call post.jar again. The output shows:
--------------------------------------
"SimplePostTool version 5.0.0
Posting files to [base] url http://localhost:8983/solr/cugna/update using content-type application/xml...
POSTing file NMLBOym_a_UN2000_10_20160511_1014.xml to [base]
SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url: http://localhost:8983/solr/cugna/update
SimplePostTool: WARNING: Response: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">400</int><int name="QTime">3</int></lst><lst name="error"><str name="msg">
Exception writing document id un_555917 to the index; possible analysis error.</str><int name="code">400</int></lst>
</response>
SimplePostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 40
0 for URL: http://localhost:8983/solr/cugna/update
1 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/cugna/update...
Time spent: 0:00:00.259"
-------------------------------------
I thought it's that doc un_555917 problem, then I comment out it in xml and do again, it keeps showing same error to every single doc. I assume there's something wrong with Solrcloud. Does anyone experience this before? What could be the problem? Can I recover it? Or I have to add all docs again?
scott.chu，scott.chu@udngroup.com
2016/5/17 (週二)
----- Original Message -----
From: scott(自己)
To: solr-user
CC:
Date: 2016/5/17 (週二) 09:39
Subject: Re(2): [scottchu] Cab I migrate solrcloud by just copying wholepackagefolder?
OK! Thanks for reminding. I'll stick to the convention.
scott.chu，scott.chu@udngroup.com
2016/5/17 (週二)
----- Original Message -----
From: Chris Hostetter
To: solr-user ; scott(自己)
CC:
Date: 2016/5/17 (週二) 02:43
Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole packagefolder?
: Message-Id: <7F...@udngroup.com>
: Subject: [scottchu] Cab I migrate solrcloud by just copying whole package
: folder?
: References:
: <DC...@udngroup.com><C162E1B1F0311FC46A64E2E2AA
: 729138@udngroup.com>,
: <85...@elyograg.org>
: In-Reply-To: <85...@elyograg.org>
https://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
-Hoss
http://www.lucidworks.com/
-----
???????????
??? AVG ?? - www.avg.com
??: 2015.0.6201 / ???: 4568/12238 - ????: 05/15/16
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12245 - 發佈日期: 05/16/16
Re(2): [scottchu] Cab I migrate solrcloud by just copying whole
packagefolder?
Posted by "scott.chu" <sc...@udngroup.com>.
OK! Thanks for reminding. I'll stick to the convention.
scott.chu，scott.chu@udngroup.com
2016/5/17 (週二)
----- Original Message -----
From: Chris Hostetter
To: solr-user ; scott(自己)
CC:
Date: 2016/5/17 (週二) 02:43
Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole packagefolder?
: Message-Id: <7F...@udngroup.com>
: Subject: [scottchu] Cab I migrate solrcloud by just copying whole package
: folder?
: References:
: <DC...@udngroup.com><C162E1B1F0311FC46A64E2E2AA
: 729138@udngroup.com>,
: <85...@elyograg.org>
: In-Reply-To: <85...@elyograg.org>
https://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
-Hoss
http://www.lucidworks.com/
-----
???????????
??? AVG ?? - www.avg.com
??: 2015.0.6201 / ???: 4568/12238 - ????: 05/15/16
Re: [scottchu] Cab I migrate solrcloud by just copying whole package
folder?
Posted by Chris Hostetter <ho...@fucit.org>.
: Message-Id: <7F...@udngroup.com>
: Subject: [scottchu] Cab I migrate solrcloud by just copying whole package
: folder?
: References:
: <DC...@udngroup.com><C162E1B1F0311FC46A64E2E2AA
: 729138@udngroup.com>,
: <85...@elyograg.org>
: In-Reply-To: <85...@elyograg.org>
https://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
-Hoss
http://www.lucidworks.com/
Re: [scottchu] Cab I migrate solrcloud by just copying whole package
folder?
Posted by Scott Chu <sc...@udngroup.com>.
Thanks to Binoy and Erick. I'll go use external zk tomorrow and do what you suggest.
Scott Chu，scott.chu@udngroup.com
2016/5/16 (週一)
----- Original Message -----
From: Erick Erickson
To: solr-user ; scott.chu
CC:
Date: 2016/5/16 (週一) 23:41
Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole package folder?
bq: How can I run that embedded zk node independently?
You don't. You run ZK independently. It _seems_ like
you should be able to copy the zk_data directory over "to
the right place" and have it found, I suspect somehow
you're not. Take a look at where it is on your source
machine and see that it's there on the destination machine....
Embedded ZK was never intended as anything other
than a way to get started, pretty soon you'll need to get
familiar with running ZK externally anyway so...
Download Zookeeper. I'm on a Mac so the commands
are slightly different, but it's just going to the Zk install
directory and './zkServer.sh start'.....
Best,
Erick
On Mon, May 16, 2016 at 8:32 AM, Scott Chu <sc...@udngroup.com> wrote:
>
> But I'm using embeded zk nodes provided by solr start command. I thought they are all under d:\solr5. How can I run that embedded zk node independently?
>
> Scott Chu，scott.chu@udngroup.com
> 2016/5/16 (週一)
> ----- Original Message -----
> From: Binoy Dalal
> To: scott.chu ; solr-user
> CC:
> Date: 2016/5/16 (週一) 23:29
> Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole package folder?
>
>
> What you copied is just the index.
> Your configurations are stored on zookeeper.
> You need to upload these to your zookeeper on your other machine and link
> it to your collection.
> Then it'll work.
>
> On Mon, 16 May 2016, 20:55 Scott Chu, <sc...@udngroup.com> wrote:
>
>> On my office pc, I install Solr 5 on d:\solr5 and create myconfigsets and
>> mynodes under it. Then run a solrcloud with 2 nodes and embedded zk nodes
>> by executing these commands:
>>
>> cd /d d:\solr5
>> bin\solr start -c -s mynode\node1
>> bin\solr start -c -s mynode\node2 -p 7973 -z localhost:9983
>> bin\solr create_collection -c cugna -d myconfigsets\cugna -shards 1
>> -replicationFactor 2
>>
>> They run well and then I insert 50k no. of docs to cugna. They are all
>> under d:\solr5 folder where I unzip solr 5 package.
>>
>> ~~~
>> Later I copy d:\solr5 to usb and bring it home. I copy it to d:\solr5 at
>> home pc(i.e. with same path). When I try to start nodes, the server log
>> shows:
>>
>> 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
>> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
>> loading config name for collection cugna
>> ...
>> 2016-05-16 14:57:18.829 INFO (qtp5592464-15) [ ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>
>> params={action=CLUSTERSTATUS&wt=json} status=500 QTime=9
>> 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
>> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
>> loading config name for collection cugna
>> ...
>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for /collections/cugna
>>
>> Does this mean Solrcloud can not migrate by copying whole Solr package?
>> How can I do it?
>>
>> Note: I want to do this because if I add all docs in lab and later when go
>> live, I wish just copy whole folder to production environment without
>> adding all docs again?
>>
> --
> Regards,
> Binoy Dalal
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12243 - 發佈日期: 05/16/16
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12243 - 發佈日期: 05/16/16
Re: [scottchu] Cab I migrate solrcloud by just copying whole package folder?
Posted by Erick Erickson <er...@gmail.com>.
bq: How can I run that embedded zk node independently?
You don't. You run ZK independently. It _seems_ like
you should be able to copy the zk_data directory over "to
the right place" and have it found, I suspect somehow
you're not. Take a look at where it is on your source
machine and see that it's there on the destination machine....
Embedded ZK was never intended as anything other
than a way to get started, pretty soon you'll need to get
familiar with running ZK externally anyway so...
Download Zookeeper. I'm on a Mac so the commands
are slightly different, but it's just going to the Zk install
directory and './zkServer.sh start'.....
Best,
Erick
On Mon, May 16, 2016 at 8:32 AM, Scott Chu <sc...@udngroup.com> wrote:
>
> But I'm using embeded zk nodes provided by solr start command. I thought they are all under d:\solr5. How can I run that embedded zk node independently?
>
> Scott Chu，scott.chu@udngroup.com
> 2016/5/16 (週一)
> ----- Original Message -----
> From: Binoy Dalal
> To: scott.chu ; solr-user
> CC:
> Date: 2016/5/16 (週一) 23:29
> Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole package folder?
>
>
> What you copied is just the index.
> Your configurations are stored on zookeeper.
> You need to upload these to your zookeeper on your other machine and link
> it to your collection.
> Then it'll work.
>
> On Mon, 16 May 2016, 20:55 Scott Chu, <sc...@udngroup.com> wrote:
>
>> On my office pc, I install Solr 5 on d:\solr5 and create myconfigsets and
>> mynodes under it. Then run a solrcloud with 2 nodes and embedded zk nodes
>> by executing these commands:
>>
>> cd /d d:\solr5
>> bin\solr start -c -s mynode\node1
>> bin\solr start -c -s mynode\node2 -p 7973 -z localhost:9983
>> bin\solr create_collection -c cugna -d myconfigsets\cugna -shards 1
>> -replicationFactor 2
>>
>> They run well and then I insert 50k no. of docs to cugna. They are all
>> under d:\solr5 folder where I unzip solr 5 package.
>>
>> ~~~
>> Later I copy d:\solr5 to usb and bring it home. I copy it to d:\solr5 at
>> home pc(i.e. with same path). When I try to start nodes, the server log
>> shows:
>>
>> 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
>> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
>> loading config name for collection cugna
>> ...
>> 2016-05-16 14:57:18.829 INFO (qtp5592464-15) [ ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>
>> params={action=CLUSTERSTATUS&wt=json} status=500 QTime=9
>> 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
>> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
>> loading config name for collection cugna
>> ...
>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for /collections/cugna
>>
>> Does this mean Solrcloud can not migrate by copying whole Solr package?
>> How can I do it?
>>
>> Note: I want to do this because if I add all docs in lab and later when go
>> live, I wish just copy whole folder to production environment without
>> adding all docs again?
>>
> --
> Regards,
> Binoy Dalal
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12243 - 發佈日期: 05/16/16
Re: [scottchu] Cab I migrate solrcloud by just copying whole package folder?
Posted by Binoy Dalal <bi...@gmail.com>.
This depends on where zookeeper is storing the data. I'm not sure exactly
how the embedded zookeeper stores data locally, but given your situation we
can be sure that it's not stored in the solr folder.
Try to find out where the embedded zookeeper stores it's data and then copy
over that directory.
That should work.
On Mon, 16 May 2016, 21:03 Scott Chu, <sc...@udngroup.com> wrote:
>
> But I'm using embeded zk nodes provided by solr start command. I thought
> they are all under d:\solr5. How can I run that embedded zk node
> independently?
>
> Scott Chu，scott.chu@udngroup.com
> 2016/5/16 (週一)
> ----- Original Message -----
> From: Binoy Dalal
> To: scott.chu ; solr-user
> CC:
> Date: 2016/5/16 (週一) 23:29
> Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole
> package folder?
>
>
> What you copied is just the index.
> Your configurations are stored on zookeeper.
> You need to upload these to your zookeeper on your other machine and link
> it to your collection.
> Then it'll work.
>
> On Mon, 16 May 2016, 20:55 Scott Chu, <sc...@udngroup.com> wrote:
>
> > On my office pc, I install Solr 5 on d:\solr5 and create myconfigsets and
> > mynodes under it. Then run a solrcloud with 2 nodes and embedded zk nodes
> > by executing these commands:
> >
> > cd /d d:\solr5
> > bin\solr start -c -s mynode\node1
> > bin\solr start -c -s mynode\node2 -p 7973 -z localhost:9983
> > bin\solr create_collection -c cugna -d myconfigsets\cugna -shards 1
> > -replicationFactor 2
> >
> > They run well and then I insert 50k no. of docs to cugna. They are all
> > under d:\solr5 folder where I unzip solr 5 package.
> >
> > ~~~
> > Later I copy d:\solr5 to usb and bring it home. I copy it to d:\solr5 at
> > home pc(i.e. with same path). When I try to start nodes, the server log
> > shows:
> >
> > 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
> > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> > loading config name for collection cugna
> > ...
> > 2016-05-16 14:57:18.829 INFO (qtp5592464-15) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>
> > params={action=CLUSTERSTATUS&wt=json} status=500 QTime=9
> > 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
> > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> > loading config name for collection cugna
> > ...
> > Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> > KeeperErrorCode = NoNode for /collections/cugna
> >
> > Does this mean Solrcloud can not migrate by copying whole Solr package?
> > How can I do it?
> >
> > Note: I want to do this because if I add all docs in lab and later when
> go
> > live, I wish just copy whole folder to production environment without
> > adding all docs again?
> >
> --
> Regards,
> Binoy Dalal
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6201 / 病毒庫: 4568/12243 - 發佈日期: 05/16/16
>
--
Regards,
Binoy Dalal
Re: [scottchu] Cab I migrate solrcloud by just copying whole package
folder?
Posted by Scott Chu <sc...@udngroup.com>.
But I'm using embeded zk nodes provided by solr start command. I thought they are all under d:\solr5. How can I run that embedded zk node independently?
Scott Chu，scott.chu@udngroup.com
2016/5/16 (週一)
----- Original Message -----
From: Binoy Dalal
To: scott.chu ; solr-user
CC:
Date: 2016/5/16 (週一) 23:29
Subject: Re: [scottchu] Cab I migrate solrcloud by just copying whole package folder?
What you copied is just the index.
Your configurations are stored on zookeeper.
You need to upload these to your zookeeper on your other machine and link
it to your collection.
Then it'll work.
On Mon, 16 May 2016, 20:55 Scott Chu, <sc...@udngroup.com> wrote:
> On my office pc, I install Solr 5 on d:\solr5 and create myconfigsets and
> mynodes under it. Then run a solrcloud with 2 nodes and embedded zk nodes
> by executing these commands:
>
> cd /d d:\solr5
> bin\solr start -c -s mynode\node1
> bin\solr start -c -s mynode\node2 -p 7973 -z localhost:9983
> bin\solr create_collection -c cugna -d myconfigsets\cugna -shards 1
> -replicationFactor 2
>
> They run well and then I insert 50k no. of docs to cugna. They are all
> under d:\solr5 folder where I unzip solr 5 package.
>
> ~~~
> Later I copy d:\solr5 to usb and bring it home. I copy it to d:\solr5 at
> home pc(i.e. with same path). When I try to start nodes, the server log
> shows:
>
> 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> loading config name for collection cugna
> ...
> 2016-05-16 14:57:18.829 INFO (qtp5592464-15) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={action=CLUSTERSTATUS&wt=json} status=500 QTime=9
> 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> loading config name for collection cugna
> ...
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /collections/cugna
>
> Does this mean Solrcloud can not migrate by copying whole Solr package?
> How can I do it?
>
> Note: I want to do this because if I add all docs in lab and later when go
> live, I wish just copy whole folder to production environment without
> adding all docs again?
>
--
Regards,
Binoy Dalal
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6201 / 病毒庫: 4568/12243 - 發佈日期: 05/16/16
Re: [scottchu] Cab I migrate solrcloud by just copying whole package folder?
Posted by Binoy Dalal <bi...@gmail.com>.
What you copied is just the index.
Your configurations are stored on zookeeper.
You need to upload these to your zookeeper on your other machine and link
it to your collection.
Then it'll work.
On Mon, 16 May 2016, 20:55 Scott Chu, <sc...@udngroup.com> wrote:
> On my office pc, I install Solr 5 on d:\solr5 and create myconfigsets and
> mynodes under it. Then run a solrcloud with 2 nodes and embedded zk nodes
> by executing these commands:
>
> cd /d d:\solr5
> bin\solr start -c -s mynode\node1
> bin\solr start -c -s mynode\node2 -p 7973 -z localhost:9983
> bin\solr create_collection -c cugna -d myconfigsets\cugna -shards 1
> -replicationFactor 2
>
> They run well and then I insert 50k no. of docs to cugna. They are all
> under d:\solr5 folder where I unzip solr 5 package.
>
> ~~~
> Later I copy d:\solr5 to usb and bring it home. I copy it to d:\solr5 at
> home pc(i.e. with same path). When I try to start nodes, the server log
> shows:
>
> 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> loading config name for collection cugna
> ...
> 2016-05-16 14:57:18.829 INFO (qtp5592464-15) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={action=CLUSTERSTATUS&wt=json} status=500 QTime=9
> 2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ]
> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> loading config name for collection cugna
> ...
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /collections/cugna
>
> Does this mean Solrcloud can not migrate by copying whole Solr package?
> How can I do it?
>
> Note: I want to do this because if I add all docs in lab and later when go
> live, I wish just copy whole folder to production environment without
> adding all docs again?
>
--
Regards,
Binoy Dalal
[scottchu] Cab I migrate solrcloud by just copying whole package
folder?
Posted by Scott Chu <sc...@udngroup.com>.
On my office pc, I install Solr 5 on d:\solr5 and create myconfigsets and mynodes under it. Then run a solrcloud with 2 nodes and embedded zk nodes by executing these commands:
cd /d d:\solr5
bin\solr start -c -s mynode\node1
bin\solr start -c -s mynode\node2 -p 7973 -z localhost:9983
bin\solr create_collection -c cugna -d myconfigsets\cugna -shards 1 -replicationFactor 2
They run well and then I insert 50k no. of docs to cugna. They are all under d:\solr5 folder where I unzip solr 5 package.
~~~
Later I copy d:\solr5 to usb and bring it home. I copy it to d:\solr5 at home pc(i.e. with same path). When I try to start nodes, the server log shows:
2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error loading config name for collection cugna
...
2016-05-16 14:57:18.829 INFO (qtp5592464-15) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=500 QTime=9
2016-05-16 14:57:18.830 ERROR (qtp5592464-15) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error loading config name for collection cugna
...
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/cugna
Does this mean Solrcloud can not migrate by copying whole Solr package? How can I do it?
Note: I want to do this because if I add all docs in lab and later when go live, I wish just copy whole folder to production environment without adding all docs again?
Re(2): [scottchu] What kind of configuration to use for this size
ofnews data?
Posted by "scott.chu" <sc...@udngroup.com>.
Originally, I want to experiment both master-slave and SolrCloud on my PC but want to save time to install another Solr server. If I have to do that, I think I have to change the default port for 2nd Solr server, right?
However, after reading the mails of Toke and Charlie. I decide to delve into SolrCloud. I'm also have no practical experience on SolrCloud. Zookeeper also looks like an unknown territory for me but it's really like a good choice for my job.
scott.chu，scott.chu@udngroup.com
2016/5/12 (週四)
----- Original Message -----
From: Shawn Heisey
To: solr-user
CC:
Date: 2016/5/11 (週三) 22:38
Subject: Re: [scottchu] What kind of configuration to use for this size ofnews data?
On 5/10/2016 10:34 PM, scott.chu wrote:
> A further question: Can master-slave and SolrCloud exist simultaneously in one Solr server? If yes, how can I do it?
No. SolrCloud uses replication internally for automated recovery on an
as-needed basis. SolrCloud completely manages multiple replicas of an
index and user-configured replication is not necessary.
I do not know what you intend with that combination, but you may want to
look into Cross-Data-Center-Replication (CDCR) in Solr 6.0.
Thanks,
Shawn
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6189 / 病毒庫: 4568/12206 - 發佈日期: 05/10/16
Re: [scottchu] What kind of configuration to use for this size of
news data?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/10/2016 10:34 PM, scott.chu wrote:
> A further question: Can master-slave and SolrCloud exist simultaneously in one Solr server? If yes, how can I do it?
No. SolrCloud uses replication internally for automated recovery on an
as-needed basis. SolrCloud completely manages multiple replicas of an
index and user-configured replication is not necessary.
I do not know what you intend with that combination, but you may want to
look into Cross-Data-Center-Replication (CDCR) in Solr 6.0.
Thanks,
Shawn
Re: [scottchu] What kind of configuration to use for this size of
news data?
Posted by "scott.chu" <sc...@udngroup.com>.
A further question: Can master-slave and SolrCloud exist simultaneously in one Solr server? If yes, how can I do it?
scott.chu，scott.chu@udngroup.com
2016/5/11 (週三)
----- Original Message -----
From: scott(自己)
To: solr-user
CC:
Date: 2016/5/11 (週三) 11:11
Subject: [scottchu] What kind of configuration to use for this size of news data?
I want to build a Solr engine for over 60-year news articles. My requests are (I use Solr 5.4.1):
1> Currently over 10M no. of docs.
2> Currently over 60GB total data size.
3> The no. of docs and data size will keep growing at the rate of 1000 no. of docs(or 8MB size) per day.
4> There are totally 5-6 different newspaper types.
My questions are:
1> Is it wokable enough just to use master-slave model? Or should I turn of SolrCloud? (I ask this due to our system management group never manage a distributed system before and they also have no knowedge of Zookeeper, shards, etc. Also they don't know how to backup/restore distributed data.)
2> Say if I choose Solrcloud anyway. I wish to keep one shard owning one specific year of data. Can it be done? What configuration should I do? (AFAIK, SolrCloud distributes data based on some intrinsic routing algorithm.)
3> If I wish to create another Solr engine with one or two particular paper types. Is it possible to copy their data directly from the big central Solr engine? Or I have to rebuild index from raw articles data? (Our business has this possibility of needs.)
I'd like to hear and use some well suggestion and experiences.
Thanks in advance and best regards.
scott.chu，scott.chu@udngroup.com
2016/5/11 (週三)