You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by 于浩 <yu...@gmail.com> on 2011/07/31 06:37:57 UTC
Solr request filter and indexing process
Hello,Dear friends,
I have got an problem in developing with solr.
In My Application ,It must sends multiple query to solr server after the
page is loaded. Then I found a problem: some request will return
statusCode:0 and QTime:0, The solr has accepted the request, but It does not
return a result document. If I send each request one by one manually ,It
will return the result. But If I send the request frequently in a very
short times, It will return nothing only statusCode:0 and QTime:0.
I think this may be a stratege for solr. but i can't find any documents or
discussions on the internet.
so i want you can help me. edited on 2011-07-28
and now I have a new problem, I am developing on php, so I connect solr
through solrPhpClient( an opensource project on google code). I find the
speed of add many documents is very slow. when I add ten documents to an
solr index, It must takes more than 5 minutes(Because of the commit process
)
anybody can help me?
Re: Solr request filter and indexing process
Posted by Michael Sokolov <so...@ifactory.com>.
The first thing that comes to mind is to check whether you are
committing after every insert. A number of things may happen when you
commit, including merges, rebuilding the spelling dictionary (is this
still true in 3.3? maybe not). It's better to commit after a batch of
inserts.
-Mike
On 7/31/2011 11:49 AM, 于浩 wrote:
> Thanks for reply ,the Solr server uses dataimporthandler, it can import 60
> thousands records(about 4G index files) properly in about 30 minutes. So I
> think the memory is enough.
> Do you know some performance factors about SolrPhpClient ?
>
>
>
> 2011/7/31 O. Klein<kl...@octoweb.nl>
>
>> I don't have much experience with Solr on windows, but have you given Solr
>> enough java heap size etc?
>>
>> Im assuming you are running Solr stand alone. Maybe try and run it in
>> Tomcat?
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Solr-request-filter-and-indexing-process-tp3212907p3213650.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
Re: Solr request filter and indexing process
Posted by 于浩 <yu...@gmail.com>.
Thanks for reply ,the Solr server uses dataimporthandler, it can import 60
thousands records(about 4G index files) properly in about 30 minutes. So I
think the memory is enough.
Do you know some performance factors about SolrPhpClient ?
2011/7/31 O. Klein <kl...@octoweb.nl>
> I don't have much experience with Solr on windows, but have you given Solr
> enough java heap size etc?
>
> Im assuming you are running Solr stand alone. Maybe try and run it in
> Tomcat?
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-request-filter-and-indexing-process-tp3212907p3213650.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr request filter and indexing process
Posted by "O. Klein" <kl...@octoweb.nl>.
I don't have much experience with Solr on windows, but have you given Solr
enough java heap size etc?
Im assuming you are running Solr stand alone. Maybe try and run it in
Tomcat?
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-request-filter-and-indexing-process-tp3212907p3213650.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr request filter and indexing process
Posted by 于浩 <yu...@gmail.com>.
thank you for reply my message. But the developer machine is a new Dell PC
with windows7 and 4G memory.
If I add 10 document through SolrPHPClient's addDocuments function with XML
format, It will takens about more than five seconds.
This speed must be unacceptable to our user.
I have already disabled the optimize operation after update, Have any other
factors is related ?
2011/7/31 O. Klein <kl...@octoweb.nl>
> Might wanna check your resources, like memory and diskspace.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-request-filter-and-indexing-process-tp3212907p3213512.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr request filter and indexing process
Posted by "O. Klein" <kl...@octoweb.nl>.
Might wanna check your resources, like memory and diskspace.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-request-filter-and-indexing-process-tp3212907p3213512.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr request filter and indexing process
Posted by 于浩 <yu...@gmail.com>.
A ha,I have found the root cause , the Solr has return the result properly
.The root cause is the SolrPHPClient, The SolrPHPClient uses
file_get_contents function for connecting to Solr by default ,this function
is not stable, usually returns http status error.
thanks for everybody who gives me help.Good luck for you!
2011/8/2 Chris Hostetter <ho...@fucit.org>
>
> : thanks for the reply. This is tomcat log files on my Solr Server:
> : I found that : if the server returns status=0 and QTime=0, the
> SolrPhpClient
> : will throughs an Exception. But the same query String will not always
> return
> : status=0 and QTime=0. The Query String is valid, I have tested them in
> Solr
>
> I know nothing about PHP but if your client code is throwing an exception
> anytime status=0 and QTime=0 then it sounds like a bug in your client code
> -- there is no reason why those two numbers being 0 should be considered
> an error. It just means the request was processed in under a millisecond.
>
>
> -Hoss
>
Re: Solr request filter and indexing process
Posted by Chris Hostetter <ho...@fucit.org>.
: thanks for the reply. This is tomcat log files on my Solr Server:
: I found that : if the server returns status=0 and QTime=0, the SolrPhpClient
: will throughs an Exception. But the same query String will not always return
: status=0 and QTime=0. The Query String is valid, I have tested them in Solr
I know nothing about PHP but if your client code is throwing an exception
anytime status=0 and QTime=0 then it sounds like a bug in your client code
-- there is no reason why those two numbers being 0 should be considered
an error. It just means the request was processed in under a millisecond.
-Hoss
Re: Solr request filter and indexing process
Posted by 于浩 <yu...@gmail.com>.
thanks for the reply. This is tomcat log files on my Solr Server:
I found that : if the server returns status=0 and QTime=0, the SolrPhpClient
will throughs an Exception. But the same query String will not always return
status=0 and QTime=0. The Query String is valid, I have tested them in Solr
Admin Page. I uses ajax to call the PHP script ,then PHP script calls
SolrPHPClient to send query request to Solr. I can't find where is the root
cause.
2011-8-1 20:41:34 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0&q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(440189)&
json.nl=map&wt=json&rows=12} hits=0 status=0 QTime=0
2011-8-1 20:41:35 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={sort=extracted_time+desc&start=0&q=((+((+京沪高铁)+++AND+(+运营)+)+((+个税)+++AND+(+改革)+))+AND+%2B(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+extracted_time:[2011-08-01T00:00:00Z-8HOURS+TO+2011-08-01T23:59:59Z-8HOURS]&
json.nl=map&wt=json&rows=3} hits=45 status=0 QTime=0
2011-8-1 20:41:35 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0&q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(622324+||+622292+||+621918)&
json.nl=map&wt=json&rows=3} hits=3 status=0 QTime=0
2011-8-1 20:41:35 org.apache.solr.core.SolrCore execute
信息: [db_operation] webapp=/solr path=/select
params={sort=extracted_time+desc&start=0&q=((+((+京沪高铁)+++AND+(+运营)+)+((+个税)+++AND+(+改革)+))+AND+-(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+-user_confirm_emotion_type:[-1+TO+-1]&
json.nl=map&wt=json&rows=9} hits=73 status=0 QTime=0
2011-8-1 20:41:36 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0&q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(621452+||+602037+||+599542+||+599541+||+599532+||+599529+||+599527+||+599526+||+599525)&
json.nl=map&wt=json&rows=9} hits=0 status=0 QTime=0
2011-8-1 20:41:37 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={sort=extracted_time+desc&start=0&q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+%2B(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:N+AND+extracted_time:[2011-08-01T00:00:00Z-8HOURS+TO+2011-08-01T23:59:59Z-8HOURS]&
json.nl=map&wt=json&rows=3} hits=136 status=0 QTime=10
2011-8-1 20:41:37 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0&q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(625436+||+625430+||+624871)&
json.nl=map&wt=json&rows=3} hits=3 status=0 QTime=0
2011-8-1 20:41:37 org.apache.solr.core.SolrCore execute
信息: [db_operation] webapp=/solr path=/select
params={sort=extracted_time+desc&start=0&q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+-(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:N+AND+-user_confirm_emotion_type:[-1+TO+-1]&
json.nl=map&wt=json&rows=9} hits=65 status=0 QTime=10
2011-8-1 20:41:37 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0&q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(624595+||+623244+||+623111+||+602402+||+597554+||+596524+||+596148+||+595638+||+595636)&
json.nl=map&wt=json&rows=9} hits=0 status=0 QTime=0
2011-8-1 20:41:39 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={sort=extracted_time+desc&start=0&q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+%2B(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:F+AND+extracted_time:[2011-08-01T00:00:00Z-8HOURS+TO+2011-08-01T23:59:59Z-8HOURS]&
json.nl=map&wt=json&rows=3} hits=83 status=0 QTime=10
2011-8-1 20:41:39 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={sort=extracted_time+desc&start=0&q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+%2B(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:B+AND+extracted_time:[2011-08-01T00:00:00Z-8HOURS+TO+2011-08-01T23:59:59Z-8HOURS]&
json.nl=map&wt=json&rows=3} hits=42 status=0 QTime=10
2011-8-1 20:41:39 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0&q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(620883+||+620765+||+620498)&
json.nl=map&wt=json&rows=3} hits=3 status=0 QTime=0
2011-8-1 20:41:39 org.apache.solr.core.SolrCore execute
信息: [db_operation] webapp=/solr path=/select
params={sort=extracted_time+desc&start=0&q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+-(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:B+AND+-user_confirm_emotion_type:[-1+TO+-1]&
json.nl=map&wt=json&rows=9} hits=47 status=0 QTime=10
2011-8-1 20:41:39 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={sort=extracted_time+desc&start=0&q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+%2B(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:M+AND+extracted_time:[2011-08-01T00:00:00Z-8HOURS+TO+2011-08-01T23:59:59Z-8HOURS]&
json.nl=map&wt=json&rows=3} hits=1 status=0 QTime=10
2011/8/1 Chris Hostetter <ho...@fucit.org>
> : page is loaded. Then I found a problem: some request will return
> : statusCode:0 and QTime:0, The solr has accepted the request, but It does
> not
> : return a result document. If I send each request one by one manually
> ,It
> : will return the result. But If I send the request frequently in a very
> : short times, It will return nothing only statusCode:0 and QTime:0.
> : I think this may be a stratege for solr. but i can't find any documents
> or
> : discussions on the internet.
>
> there is no stratgy like whta you describe in solr -- requests should not
> behave differnetly depending on wether you send them one at a time, or if
> you send them all concurrently -- many parallel requests may bog down the
> server, causing the responses to take longer, but you should not see
> different results from solr.
>
> you have not relaly given us much information at all to go on to make
> suggestsion about the problem you are seeing -- at a minimum some server
> log showing the types of requests you are making and whta *solr* says
> about those requests in it's logs is neccesary, but it may also be helpful
> if you show us the client code you are using to make the requests. My
> suspicioun is that Solr isn't recieving the requests you think you are
> sending.
>
> Please review this wiki page carefully before sending any replies...
>
> http://wiki.apache.org/solr/UsingMailingLists
>
>
> -Hoss
>
Re: Solr request filter and indexing process
Posted by Chris Hostetter <ho...@fucit.org>.
: page is loaded. Then I found a problem: some request will return
: statusCode:0 and QTime:0, The solr has accepted the request, but It does not
: return a result document. If I send each request one by one manually ,It
: will return the result. But If I send the request frequently in a very
: short times, It will return nothing only statusCode:0 and QTime:0.
: I think this may be a stratege for solr. but i can't find any documents or
: discussions on the internet.
there is no stratgy like whta you describe in solr -- requests should not
behave differnetly depending on wether you send them one at a time, or if
you send them all concurrently -- many parallel requests may bog down the
server, causing the responses to take longer, but you should not see
different results from solr.
you have not relaly given us much information at all to go on to make
suggestsion about the problem you are seeing -- at a minimum some server
log showing the types of requests you are making and whta *solr* says
about those requests in it's logs is neccesary, but it may also be helpful
if you show us the client code you are using to make the requests. My
suspicioun is that Solr isn't recieving the requests you think you are
sending.
Please review this wiki page carefully before sending any replies...
http://wiki.apache.org/solr/UsingMailingLists
-Hoss