You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@lucene.apache.org by salma begumshaik <sa...@infosys.com> on 2011/10/12 06:33:27 UTC
Lucene search -- Concurrent users issue.. If more than 300 users
are using at a time search results are getting delayed
Hi
I am using lucene-core-3.0.3.jar for searching. I have implemented the code
as exactly mentioned in Lucene.in.Action.2nd.Edition.pdf.
I am using Tika0.9.jar to index the web pages.
For one user it is working perfectly and getting results within 1to 2 secs.
but when concurrent users for ex 350 users are using the search, results are
getting delayed and it is taking more than 36 seconds.
1) Is there any possible way to handle concurrency issue.
2) Is there any configuration to mention concurrent users.
Thanks in advance
Salma.
--
View this message in context: http://lucene.472066.n3.nabble.com/Lucene-search-Concurrent-users-issue-If-more-than-300-users-are-using-at-a-time-search-results-are-gd-tp3414779p3414779.html
Sent from the Lucene - General mailing list archive at Nabble.com.
Re: Lucene search -- Concurrent users issue.. If more than 300
users are using at a time search results are getting delayed
Posted by salma begumshaik <sa...@infosys.com>.
I have copied the code.. please refer to that and tell me what optimization
to be done.
how many documents you have in your index -- we have hardly 600 web pages.
Eventually it'd be interesting what hardware you are using -- our
production server is linux..
--
View this message in context: http://lucene.472066.n3.nabble.com/Lucene-search-Concurrent-users-issue-If-more-than-300-users-are-using-at-a-time-search-results-are-gd-tp3414779p3415031.html
Sent from the Lucene - General mailing list archive at Nabble.com.
Re: Lucene search -- Concurrent users issue.. If more than 300
users are using at a time search results are getting delayed
Posted by salma begumshaik <sa...@infosys.com>.
Hi Simon
thank you for the reply.
How do we keep searcher open. for each user this application will be
executed right. so for each user FSDirectory will be open and then close it.
can you please explain in detail.
all code will be written in java class. when user is searching is something,
this class will get invoked and results will be returned.
I am facing issue with concurrent users.
thanks in advance,
salma
--
View this message in context: http://lucene.472066.n3.nabble.com/Lucene-search-Concurrent-users-issue-If-more-than-300-users-are-using-at-a-time-search-results-are-gd-tp3414779p3415407.html
Sent from the Lucene - General mailing list archive at Nabble.com.
Re: Lucene search -- Concurrent users issue.. If more than 300 users
are using at a time search results are getting delayed
Posted by Simon Willnauer <si...@googlemail.com>.
alright :)
This is going to kill your app :)
Directory dir = FSDirectory.open(new File(indexDir));
IndexSearcher is = new IndexSearcher(dir);
you should try to keep this searcher open as long as possible and
reuse it. Its threadsafe so you can simply pass it to all your
searches.
Once you commit your IndexWriter you need to reopen this searcher and
close the old one. This should solve your perf issues.
simon
On Wed, Oct 12, 2011 at 9:24 AM, salma begumshaik
<sa...@infosys.com> wrote:
>
>
> public class Searcher {
>
> private int hitCounts = 0;
> private ResultDocument[] scoreDocs = null;
> public Searcher() {
> }
>
> public void search1(String searchValue, String isArchive)
> throws IllegalArgumentException, IOException, ParseException {
>
> String[] args = new String[2];
> ServletContext contxt =
> org.apache.struts2.ServletActionContext.getServletContext();
> if(isArchive.equalsIgnoreCase("true")){
> args[0] = contxt.getRealPath("archiveindexes");
> }
> else{
> args[0] = contxt.getRealPath("indexes");
> }
> args[1] = searchValue;
> String indexDir = args[0];
> String q = args[1];
>
> search(indexDir, q);
> }
>
> public void search(String indexDir, String q) throws IOException,
> ParseException {
> Directory dir = FSDirectory.open(new File(indexDir));
> IndexSearcher is = new IndexSearcher(dir);
> QueryParser parser1 = new QueryParser(Version.LUCENE_30, "contents",new
> StandardAnalyzer(Version.LUCENE_30));
> Query query1 = parser1.parse(q + "*");
> long start1 = System.currentTimeMillis();
> TopDocs hits1 = is.search(query1, 20);
> hitCounts = hits1.totalHits;
> hits1 = is.search(query1, hitCounts);
> long end1 = System.currentTimeMillis();
> System.err.println("Found " + hits1.totalHits + " document(s) (in "
> + (end1 - start1) + " milliseconds) that matched query1 '" + q+ "':");
>
> int totalLength = hits1.scoreDocs.length;
> scoreDocs = new ResultDocument[totalLength];
> int i=0;
> for (ScoreDoc scoreDoc1 : hits1.scoreDocs) {
> Document doc = is.doc(scoreDoc1.doc);
> ResultDocument resultDoc = new ResultDocument(doc);
> scoreDocs[i++]=resultDoc;
> }
> is.close();
> }
>
> public int getHitCounts() {
> return hitCounts;
> }
>
> public ResultDocument[] getScoreDocs() {
> return scoreDocs;
> }
> }
>
>
> -----------Searching class----------------
> /*
> * To change this template, choose Tools | Templates
> * and open the template in the editor.
> */
>
> package com.infy.stateportal.portalsearch.search;
>
> import java.io.IOException;
>
> import javax.servlet.http.HttpServletRequest;
> import javax.servlet.http.HttpSession;
>
> import org.apache.lucene.queryParser.ParseException;
>
> import com.infy.stateportal.portalsearch.search.ResultDocument;
> import com.infy.stateportal.portalsearch.search.Searcher;
> import com.opensymphony.xwork2.ActionSupport;
>
> public class Searching extends ActionSupport {
>
> private static final long serialVersionUID = 1L;
> private String message;
> private String searchContent;
> private int start=0;
> private int end=0;
> private int current =0;
> private ResultDocument[] resultDocument = null;
> private ResultDocument[] pageResultDocument = null;
> private int hitCount=0;
> private int resultHitsPerPage = 10;
>
> public Searching() {
> }
>
> public void validate() {
>
> HttpServletRequest req =
> org.apache.struts2.ServletActionContext.getRequest();
> if(req.getParameter("searchContent") !=null){
> setSearchContent(req.getParameter("searchContent").toString());
> }
> if (getSearchContent().length() == 0) {
> addFieldError("searchContent", "Search content is required");
> }
> }
>
> public String execute() {
> try {
> if(getSearchContent()==null || getSearchContent()=="" ||
> (getSearchContent()!=null
> && getSearchContent().equalsIgnoreCase(""))) {
>
> }
> HttpServletRequest req =
> org.apache.struts2.ServletActionContext.getRequest();
> if(req.getParameter("current")!=null) {
>
> if(req.getParameter("start")!=null) {
> //System.out.println("start "+req.getParameter("start"));
> start = Integer.parseInt(req.getParameter("start"));
> }
> if(req.getParameter("end")!=null) {
> //System.out.println("end "+req.getParameter("end"));
> end = Integer.parseInt(req.getParameter("end"));
> }
> if(req.getParameter("current")!=null) {
> //System.out.println("current "+req.getParameter("current"));
> current = Integer.parseInt(req.getParameter("current"));
> }
> HttpSession session = req.getSession();
> resultDocument =
> (ResultDocument[])session.getAttribute("searchResult");
> hitCount = (Integer)session.getAttribute("hitCount");
> if(current >5) {
> start = current -5;
> end = ((current +5)<(getPageCount()))?(current+5):(getPageCount());
> }
> if(current<=5) {
> start = 0;
> end = (getPageCount()>10)?10:getPageCount();
> }
> //if(current )
> }
> else {
> String isArchive = "";
> if(req.getParameter("isArchive")!=null){
> isArchive = (req.getParameter("isArchive")).toString();
> }
> Searcher searcher = new Searcher();
> searcher.search1(getSearchContent(), isArchive);
> //searcher.search(getSearchContent());
> resultDocument = searcher.getScoreDocs();
> hitCount = searcher.getHitCounts();
> HttpSession session = req.getSession(true);
> session.setAttribute("searchResult", resultDocument);
> session.setAttribute("hitCount", hitCount);
> end = getPageCount();
> if(getPageCount()>10) {
> end = 10;
> }
> }
> pageResultDocument = new ResultDocument[resultHitsPerPage];
> int j=0;
> for(int i=(current)*resultHitsPerPage;(i<resultDocument.length) &&
> (j<resultHitsPerPage);i++) {
> pageResultDocument[j++]=resultDocument[i];
> }
> //System.out.println(" searcher.getHitCounts() "+getPageCount());
> //current++;
>
> } catch (IllegalArgumentException e) {
> e.printStackTrace();
> } catch (IOException e) {
> e.printStackTrace();
> } catch (ParseException e) {
> e.printStackTrace();
> }
> setMessage("Hello " + getSearchContent());
> return "SUCCESS";
> }
>
> /**
> * @return the message
> */
> public String getMessage() {
> return message;
> }
>
> /**
> * @param message the message to set
> */
> public void setMessage(String message) {
> this.message = message;
> }
>
> /**
> * @return the userName
> */
> public String getSearchContent() {
> return searchContent;
> }
>
> /**
> * @param userName the userName to set
> */
> public void setSearchContent(String searchContent) {
> this.searchContent = searchContent;
> }
>
> public int getHitCount() {
> return hitCount;
> }
>
> public ResultDocument[] getScoreDocs() {
> return this.pageResultDocument;
> }
>
> public int getStart() {
> return start;
> }
>
> public int getEnd() {
> return end;
> }
>
> private int getPageCount() {
> return
> ((hitCount%resultHitsPerPage)==0)?(hitCount/resultHitsPerPage):(hitCount/resultHitsPerPage)+1;
> }
>
> public int getCurrent() {
> return current;
> }
>
> }
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Lucene-search-Concurrent-users-issue-If-more-than-300-users-are-using-at-a-time-search-results-are-gd-tp3414779p3415008.html
> Sent from the Lucene - General mailing list archive at Nabble.com.
>
Re: Lucene search -- Concurrent users issue.. If more than 300
users are using at a time search results are getting delayed
Posted by salma begumshaik <sa...@infosys.com>.
public class Searcher {
private int hitCounts = 0;
private ResultDocument[] scoreDocs = null;
public Searcher() {
}
public void search1(String searchValue, String isArchive)
throws IllegalArgumentException, IOException, ParseException {
String[] args = new String[2];
ServletContext contxt =
org.apache.struts2.ServletActionContext.getServletContext();
if(isArchive.equalsIgnoreCase("true")){
args[0] = contxt.getRealPath("archiveindexes");
}
else{
args[0] = contxt.getRealPath("indexes");
}
args[1] = searchValue;
String indexDir = args[0];
String q = args[1];
search(indexDir, q);
}
public void search(String indexDir, String q) throws IOException,
ParseException {
Directory dir = FSDirectory.open(new File(indexDir));
IndexSearcher is = new IndexSearcher(dir);
QueryParser parser1 = new QueryParser(Version.LUCENE_30, "contents",new
StandardAnalyzer(Version.LUCENE_30));
Query query1 = parser1.parse(q + "*");
long start1 = System.currentTimeMillis();
TopDocs hits1 = is.search(query1, 20);
hitCounts = hits1.totalHits;
hits1 = is.search(query1, hitCounts);
long end1 = System.currentTimeMillis();
System.err.println("Found " + hits1.totalHits + " document(s) (in "
+ (end1 - start1) + " milliseconds) that matched query1 '" + q+ "':");
int totalLength = hits1.scoreDocs.length;
scoreDocs = new ResultDocument[totalLength];
int i=0;
for (ScoreDoc scoreDoc1 : hits1.scoreDocs) {
Document doc = is.doc(scoreDoc1.doc);
ResultDocument resultDoc = new ResultDocument(doc);
scoreDocs[i++]=resultDoc;
}
is.close();
}
public int getHitCounts() {
return hitCounts;
}
public ResultDocument[] getScoreDocs() {
return scoreDocs;
}
}
-----------Searching class----------------
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
package com.infy.stateportal.portalsearch.search;
import java.io.IOException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpSession;
import org.apache.lucene.queryParser.ParseException;
import com.infy.stateportal.portalsearch.search.ResultDocument;
import com.infy.stateportal.portalsearch.search.Searcher;
import com.opensymphony.xwork2.ActionSupport;
public class Searching extends ActionSupport {
private static final long serialVersionUID = 1L;
private String message;
private String searchContent;
private int start=0;
private int end=0;
private int current =0;
private ResultDocument[] resultDocument = null;
private ResultDocument[] pageResultDocument = null;
private int hitCount=0;
private int resultHitsPerPage = 10;
public Searching() {
}
public void validate() {
HttpServletRequest req =
org.apache.struts2.ServletActionContext.getRequest();
if(req.getParameter("searchContent") !=null){
setSearchContent(req.getParameter("searchContent").toString());
}
if (getSearchContent().length() == 0) {
addFieldError("searchContent", "Search content is required");
}
}
public String execute() {
try {
if(getSearchContent()==null || getSearchContent()=="" ||
(getSearchContent()!=null
&& getSearchContent().equalsIgnoreCase(""))) {
}
HttpServletRequest req =
org.apache.struts2.ServletActionContext.getRequest();
if(req.getParameter("current")!=null) {
if(req.getParameter("start")!=null) {
//System.out.println("start "+req.getParameter("start"));
start = Integer.parseInt(req.getParameter("start"));
}
if(req.getParameter("end")!=null) {
//System.out.println("end "+req.getParameter("end"));
end = Integer.parseInt(req.getParameter("end"));
}
if(req.getParameter("current")!=null) {
//System.out.println("current "+req.getParameter("current"));
current = Integer.parseInt(req.getParameter("current"));
}
HttpSession session = req.getSession();
resultDocument =
(ResultDocument[])session.getAttribute("searchResult");
hitCount = (Integer)session.getAttribute("hitCount");
if(current >5) {
start = current -5;
end = ((current +5)<(getPageCount()))?(current+5):(getPageCount());
}
if(current<=5) {
start = 0;
end = (getPageCount()>10)?10:getPageCount();
}
//if(current )
}
else {
String isArchive = "";
if(req.getParameter("isArchive")!=null){
isArchive = (req.getParameter("isArchive")).toString();
}
Searcher searcher = new Searcher();
searcher.search1(getSearchContent(), isArchive);
//searcher.search(getSearchContent());
resultDocument = searcher.getScoreDocs();
hitCount = searcher.getHitCounts();
HttpSession session = req.getSession(true);
session.setAttribute("searchResult", resultDocument);
session.setAttribute("hitCount", hitCount);
end = getPageCount();
if(getPageCount()>10) {
end = 10;
}
}
pageResultDocument = new ResultDocument[resultHitsPerPage];
int j=0;
for(int i=(current)*resultHitsPerPage;(i<resultDocument.length) &&
(j<resultHitsPerPage);i++) {
pageResultDocument[j++]=resultDocument[i];
}
//System.out.println(" searcher.getHitCounts() "+getPageCount());
//current++;
} catch (IllegalArgumentException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (ParseException e) {
e.printStackTrace();
}
setMessage("Hello " + getSearchContent());
return "SUCCESS";
}
/**
* @return the message
*/
public String getMessage() {
return message;
}
/**
* @param message the message to set
*/
public void setMessage(String message) {
this.message = message;
}
/**
* @return the userName
*/
public String getSearchContent() {
return searchContent;
}
/**
* @param userName the userName to set
*/
public void setSearchContent(String searchContent) {
this.searchContent = searchContent;
}
public int getHitCount() {
return hitCount;
}
public ResultDocument[] getScoreDocs() {
return this.pageResultDocument;
}
public int getStart() {
return start;
}
public int getEnd() {
return end;
}
private int getPageCount() {
return
((hitCount%resultHitsPerPage)==0)?(hitCount/resultHitsPerPage):(hitCount/resultHitsPerPage)+1;
}
public int getCurrent() {
return current;
}
}
--
View this message in context: http://lucene.472066.n3.nabble.com/Lucene-search-Concurrent-users-issue-If-more-than-300-users-are-using-at-a-time-search-results-are-gd-tp3414779p3415008.html
Sent from the Lucene - General mailing list archive at Nabble.com.
Re: Lucene search -- Concurrent users issue.. If more than 300 users
are using at a time search results are getting delayed
Posted by Simon Willnauer <si...@googlemail.com>.
hey,
concurrency should not be an issue for lucene. Can you provide some
more information on what you are doing?
1 to 2 seconds sounds very very long for a single user. Are you doing
something like opening the indexreader for each request?
I'd also be interested in what kind of queries you are executing and
how many documents you have in your index....
Eventually it'd be interesting what hardware you are using.
simon
On Wed, Oct 12, 2011 at 6:33 AM, salma begumshaik
<sa...@infosys.com> wrote:
> Hi
>
> I am using lucene-core-3.0.3.jar for searching. I have implemented the code
> as exactly mentioned in Lucene.in.Action.2nd.Edition.pdf.
>
> I am using Tika0.9.jar to index the web pages.
>
> For one user it is working perfectly and getting results within 1to 2 secs.
> but when concurrent users for ex 350 users are using the search, results are
> getting delayed and it is taking more than 36 seconds.
>
> 1) Is there any possible way to handle concurrency issue.
> 2) Is there any configuration to mention concurrent users.
>
> Thanks in advance
> Salma.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Lucene-search-Concurrent-users-issue-If-more-than-300-users-are-using-at-a-time-search-results-are-gd-tp3414779p3414779.html
> Sent from the Lucene - General mailing list archive at Nabble.com.
>