You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by neeraj shah <ne...@gmail.com> on 2013/06/22 08:07:07 UTC
how to fetch qyery feild + other feild related results in Lucene3.6
Hello,
Im using Lucene 3.6.2. and my file, which i indexed ,is something like
this :
FIELD-1 FIELD-2 FIELD-3 FIELD-4
DOC1: A 1111 2222 ABC
DOC2: B 1111 KKKK ABC
DOC3: C 1111 JJJJ ABC
DOC4: D 1111 LLLL ABC
DOC5: E 1111 WWW ABC
DOC6: F 2222 CCCC DEF
DOC7: G 2222 SSSS DEF
DOC8: H 2222 FFFF DEF
DOC9: I 2222 QQQQ DEF
now if i search over field "FIELD-3" and search for "222" ,i would get the
single result of "ABC".
My requirement is i want to fetch all results which contains "ABC". while
quering Field-4 value is not knwon so i can not put direct "ABC" for search.
how can i do this?
RE: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by Uwe Schindler <uw...@thetaphi.de>.
Tom e it looks like you want to try the Join functionality:
http://lucene.apache.org/core/3_6_2/api/all/org/apache/lucene/search/join/package-summary.html
It is not completely flexible, but should be able to do what you describe.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: neeraj shah [mailto:neerajshah84@gmail.com]
> Sent: Saturday, June 22, 2013 11:39 AM
> To: java-user@lucene.apache.org
> Subject: Re: how to fetch qyery feild + other feild related results in Lucene3.6
>
> i have already tried this. I am fetching the info of "ABC" and then passing
> then again in searcher with new Query then adding the required result.
> but this method taking very very long time (more than 100 secs). my data file
> is very big size would be around 137MB.
> So was asking is there another way to get the result in Lucene.
>
>
> On Sat, Jun 22, 2013 at 2:57 PM, Roberto Ragusa
> <ma...@robertoragusa.it>wrote:
>
> > On 06/22/2013 08:07 AM, neeraj shah wrote:
> > > Hello,
> > >
> > > Im using Lucene 3.6.2. and my file, which i indexed ,is something
> > > like this :
> > >
> > > FIELD-1 FIELD-2 FIELD-3 FIELD-4
> > > DOC1: A 1111 2222 ABC
> > > DOC2: B 1111 KKKK ABC
> > > DOC3: C 1111 JJJJ ABC
> > > DOC4: D 1111 LLLL ABC
> > > DOC5: E 1111 WWW ABC
> > > DOC6: F 2222 CCCC DEF
> > > DOC7: G 2222 SSSS DEF
> > > DOC8: H 2222 FFFF DEF
> > > DOC9: I 2222 QQQQ DEF
> > >
> > > now if i search over field "FIELD-3" and search for "222" ,i would
> > > get
> > the
> > > single result of "ABC".
> > > My requirement is i want to fetch all results which contains "ABC".
> > > while quering Field-4 value is not knwon so i can not put direct
> > > "ABC" for
> > search.
> > >
> > > how can i do this?
> >
> > Two searches?
> > Search 2222, identify doc1, get the stored field4 "ABC", now search
> > for "field4:ABC".
> > If the first search returns many results, I suppose you may want to
> > add them all to the second search.
> >
> > --
> > Roberto Ragusa mail at robertoragusa.it
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> --
> With Regards,
> Neeraj Kumar Shah
> +919819474146
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by neeraj shah <ne...@gmail.com>.
this is the way im indexing the file:
FileInputStream fr = new FileInputStream(file);
BufferedInputStream bfr = new BufferedInputStream(fr);
DataInputStream dbfr = new DataInputStream(bfr);
while(dbfr.available()!=0){
String line = dbfr.readLine();
if(line!=null){
String[] id_name = line.split("\t");
//Mind RUB_Mind_1015181 Mind / DAY_TEST 6 bism 1
RUB_Mind_1015181 REP_Boenning
String section = id_name[0];
String id = id_name[1];
String name = id_name[2];
System.out.println(name);
//String remCount = id_name[3];
String remname = id_name[3];
String remmarks = id_name[4];
String remrid = id_name[5];
String remauthid = id_name[6];
String rubnum = remrid.substring(remrid.lastIndexOf("_")+1,
remrid.length());
//String content =
section+"\t"+name+"\t"+remname+"\t"+remmarks+"\t"+remauthid;
Document doc = new Document();
//doc.add(new
Field("contents",content,Field.Store.COMPRESS,Field.Index.ANALYZED));
doc.add(new
Field("contents",name,Field.Store.YES,Field.Index.ANALYZED));
doc.add(new
Field("title",id,Field.Store.YES,Field.Index.ANALYZED));
doc.add(new
Field("section",section,Field.Store.YES,Field.Index.ANALYZED));
//doc.add(new
Field("remcount",remCount,Field.Store.YES,Field.Index.NOT_ANALYZED));
doc.add(new
Field("remname",remname,Field.Store.YES,Field.Index.ANALYZED));
doc.add(new
Field("remmarks",remmarks,Field.Store.YES,Field.Index.ANALYZED));
doc.add(new
Field("remrid",remrid,Field.Store.YES,Field.Index.ANALYZED));
doc.add(new
Field("remauthid",remauthid,Field.Store.YES,Field.Index.ANALYZED));
doc.add(new
Field("rubnum",rubnum,Field.Store.YES,Field.Index.ANALYZED));
writer.addDocument(doc);
}
}
On Mon, Jun 24, 2013 at 2:31 PM, neeraj shah <ne...@gmail.com> wrote:
> so how can i solve this and reduce time?
>
>
> On Mon, Jun 24, 2013 at 2:21 PM, Roberto Ragusa <ma...@robertoragusa.it>wrote:
>
>> On 06/24/2013 08:38 AM, neeraj shah wrote:
>> > My hit size is 127674 and even if i comment the remedy fetching code (
>> the
>> > second search in for loop) still its taking very long time.
>> > This is the code which im using without Remedy fetching code :
>> >
>> >
>> > for(int k=0;k<arrScoreDoc.length;k++){
>> > Document doc = searcher.doc(arrScoreDoc[k].doc);
>> > String remrid = doc.get("remrid");
>> >
>> > SearchHit o = new SearchHit();
>> > String contents = doc.get("contents");
>> > String section = doc.get("section");
>> >
>> > o.setContents(contents.replaceAll("/", " ; "));
>> > o.setSection(section);
>> > o.setRemrid(remrid);
>> >
>> >
>> > }
>>
>> You are doing 3 doc.get per result; those 400,000 calls are killing you.
>> Do you see disk activity or is it entirely cached?
>> What operating system are you running on?
>>
>>
>> --
>> Roberto Ragusa mail at robertoragusa.it
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> With Regards,
> Neeraj Kumar Shah
> +919819474146
>
--
With Regards,
Neeraj Kumar Shah
+919819474146
Re: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by neeraj shah <ne...@gmail.com>.
so how can i solve this and reduce time?
On Mon, Jun 24, 2013 at 2:21 PM, Roberto Ragusa <ma...@robertoragusa.it>wrote:
> On 06/24/2013 08:38 AM, neeraj shah wrote:
> > My hit size is 127674 and even if i comment the remedy fetching code (
> the
> > second search in for loop) still its taking very long time.
> > This is the code which im using without Remedy fetching code :
> >
> >
> > for(int k=0;k<arrScoreDoc.length;k++){
> > Document doc = searcher.doc(arrScoreDoc[k].doc);
> > String remrid = doc.get("remrid");
> >
> > SearchHit o = new SearchHit();
> > String contents = doc.get("contents");
> > String section = doc.get("section");
> >
> > o.setContents(contents.replaceAll("/", " ; "));
> > o.setSection(section);
> > o.setRemrid(remrid);
> >
> >
> > }
>
> You are doing 3 doc.get per result; those 400,000 calls are killing you.
> Do you see disk activity or is it entirely cached?
> What operating system are you running on?
>
>
> --
> Roberto Ragusa mail at robertoragusa.it
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
With Regards,
Neeraj Kumar Shah
+919819474146
Re: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by Roberto Ragusa <ma...@robertoragusa.it>.
On 06/24/2013 08:38 AM, neeraj shah wrote:
> My hit size is 127674 and even if i comment the remedy fetching code ( the
> second search in for loop) still its taking very long time.
> This is the code which im using without Remedy fetching code :
>
>
> for(int k=0;k<arrScoreDoc.length;k++){
> Document doc = searcher.doc(arrScoreDoc[k].doc);
> String remrid = doc.get("remrid");
>
> SearchHit o = new SearchHit();
> String contents = doc.get("contents");
> String section = doc.get("section");
>
> o.setContents(contents.replaceAll("/", " ; "));
> o.setSection(section);
> o.setRemrid(remrid);
>
>
> }
You are doing 3 doc.get per result; those 400,000 calls are killing you.
Do you see disk activity or is it entirely cached?
What operating system are you running on?
--
Roberto Ragusa mail at robertoragusa.it
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by neeraj shah <ne...@gmail.com>.
My hit size is 127674 and even if i comment the remedy fetching code ( the
second search in for loop) still its taking very long time.
This is the code which im using without Remedy fetching code :
StandardAnalyzer analyzer = new
StandardAnalyzer(Version.LUCENE_36);
IndexReader reader = IndexReader.open(FSDirectory.open(new
File("C:/hierarchy/idname/index")));
final IndexSearcher searcher = new IndexSearcher(reader);
Query qu = new QueryParser(Version.LUCENE_36,"remname",
analyzer).parse(query);
//Sort sort = new Sort(new
SortField("contents",SortField.STRING));
TopDocs hits = searcher.search(qu,searcher.maxDoc());
final ScoreDoc[] arrScoreDoc = hits.scoreDocs;
System.out.println("hit size: "+arrScoreDoc.length);
for(int k=0;k<arrScoreDoc.length;k++){
Document doc = searcher.doc(arrScoreDoc[k].doc);
String remrid = doc.get("remrid");
SearchHit o = new SearchHit();
String contents = doc.get("contents");
String section = doc.get("section");
o.setContents(contents.replaceAll("/", " ; "));
o.setSection(section);
o.setRemrid(remrid);
}
On Mon, Jun 24, 2013 at 12:01 PM, neeraj shah <ne...@gmail.com>wrote:
>
> its 137MB only.
>
> my code id is:
>
> StandardAnalyzer analyzer = new
> StandardAnalyzer(Version.LUCENE_36);
> IndexReader reader = IndexReader.open(FSDirectory.open(new
> File("C:/hierarchy/idname/index")));
> IndexSearcher searcher = new IndexSearcher(reader);
> Query qu = new QueryParser(Version.LUCENE_36,"remname",
> analyzer).parse(query);
>
> TopDocs hits = searcher.search(qu,searcher.maxDoc());
> final ScoreDoc[] arrScoreDoc = hits.scoreDocs;
> System.out.println("hit size: "+arrScoreDoc.length);
>
> List lstRubricIdsAdded = new ArrayList();
>
> for(int k=0;k<arrScoreDoc.length;k++){
> Document doc = searcher.doc(arrScoreDoc[k].doc);
> String remrid = doc.get("remrid");
>
> if(!lstRubricIdsAdded.contains(remrid)){
> lstRubricIdsAdded.add(remrid);
>
> SearchHit o = new SearchHit();
> String contents = doc.get("contents");
> String section = doc.get("section");
>
> o.setContents(contents.replaceAll("/", " ; "));
> o.setSection(section);
> o.setRemrid(remrid);
>
> Query q = new QueryParser(Version.LUCENE_36,"remrid",
> analyzer).parse(remrid);
> TopDocs hitsRubId = searcher.search(q,searcher.maxDoc());
> ScoreDoc[] arrRubIdScoreDoc = hitsRubId.scoreDocs;
>
> for(int h=0;h<arrRubIdScoreDoc.length;h++){
> Document docRem =
> searcher.doc(arrRubIdScoreDoc[h].doc);
>
> String remmarks = docRem.get("remmarks");
> String remauthid = docRem.get("remauthid");
> String remname = doc.get("remname");
>
> Remedy remedy = new Remedy(remname);
>
> remedy.setRubricID(remrid);
> remedy.setAuhtorId(remauthid);
> int mark = Integer.parseInt(remmarks);
> remedy.setMarks(mark);
>
>
> remedy.setAuhtorAbbrivation(auth.get(remauthid.toLowerCase().trim()).toString());
>
> o.getRemedyInList().add(remedy);
>
> }
>
> }
> }
>
> this code is taking very long time.
>
>
> On Sat, Jun 22, 2013 at 6:52 PM, Roberto Ragusa <ma...@robertoragusa.it>wrote:
>
>> On 06/22/2013 11:38 AM, neeraj shah wrote:
>> > i have already tried this. I am fetching the info of "ABC" and then
>> passing
>> > then again in searcher with new Query then adding the required result.
>> > but this method taking very very long time (more than 100 secs). my data
>> > file is very big size would be around 137MB.
>> > So was asking is there another way to get the result in Lucene.
>>
>> 100 secs?
>> And you are calling 137MB a very big size?
>>
>> Something doesn't feel right.
>>
>> Are you doing two searches (1+1) or a lot of them (1+N)?
>> I really reread your mail a few times to understand if I had misread
>> seconds for milliseconds or MB for GB...
>>
>> --
>> Roberto Ragusa mail at robertoragusa.it
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> With Regards,
> Neeraj Kumar Shah
> +919819474146
>
--
With Regards,
Neeraj Kumar Shah
+919819474146
Re: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by neeraj shah <ne...@gmail.com>.
its 137MB only.
my code id is:
StandardAnalyzer analyzer = new
StandardAnalyzer(Version.LUCENE_36);
IndexReader reader = IndexReader.open(FSDirectory.open(new
File("C:/hierarchy/idname/index")));
IndexSearcher searcher = new IndexSearcher(reader);
Query qu = new QueryParser(Version.LUCENE_36,"remname",
analyzer).parse(query);
TopDocs hits = searcher.search(qu,searcher.maxDoc());
final ScoreDoc[] arrScoreDoc = hits.scoreDocs;
System.out.println("hit size: "+arrScoreDoc.length);
List lstRubricIdsAdded = new ArrayList();
for(int k=0;k<arrScoreDoc.length;k++){
Document doc = searcher.doc(arrScoreDoc[k].doc);
String remrid = doc.get("remrid");
if(!lstRubricIdsAdded.contains(remrid)){
lstRubricIdsAdded.add(remrid);
SearchHit o = new SearchHit();
String contents = doc.get("contents");
String section = doc.get("section");
o.setContents(contents.replaceAll("/", " ; "));
o.setSection(section);
o.setRemrid(remrid);
Query q = new QueryParser(Version.LUCENE_36,"remrid",
analyzer).parse(remrid);
TopDocs hitsRubId = searcher.search(q,searcher.maxDoc());
ScoreDoc[] arrRubIdScoreDoc = hitsRubId.scoreDocs;
for(int h=0;h<arrRubIdScoreDoc.length;h++){
Document docRem = searcher.doc(arrRubIdScoreDoc[h].doc);
String remmarks = docRem.get("remmarks");
String remauthid = docRem.get("remauthid");
String remname = doc.get("remname");
Remedy remedy = new Remedy(remname);
remedy.setRubricID(remrid);
remedy.setAuhtorId(remauthid);
int mark = Integer.parseInt(remmarks);
remedy.setMarks(mark);
remedy.setAuhtorAbbrivation(auth.get(remauthid.toLowerCase().trim()).toString());
o.getRemedyInList().add(remedy);
}
}
}
this code is taking very long time.
On Sat, Jun 22, 2013 at 6:52 PM, Roberto Ragusa <ma...@robertoragusa.it>wrote:
> On 06/22/2013 11:38 AM, neeraj shah wrote:
> > i have already tried this. I am fetching the info of "ABC" and then
> passing
> > then again in searcher with new Query then adding the required result.
> > but this method taking very very long time (more than 100 secs). my data
> > file is very big size would be around 137MB.
> > So was asking is there another way to get the result in Lucene.
>
> 100 secs?
> And you are calling 137MB a very big size?
>
> Something doesn't feel right.
>
> Are you doing two searches (1+1) or a lot of them (1+N)?
> I really reread your mail a few times to understand if I had misread
> seconds for milliseconds or MB for GB...
>
> --
> Roberto Ragusa mail at robertoragusa.it
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
With Regards,
Neeraj Kumar Shah
+919819474146
Re: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by Roberto Ragusa <ma...@robertoragusa.it>.
On 06/22/2013 11:38 AM, neeraj shah wrote:
> i have already tried this. I am fetching the info of "ABC" and then passing
> then again in searcher with new Query then adding the required result.
> but this method taking very very long time (more than 100 secs). my data
> file is very big size would be around 137MB.
> So was asking is there another way to get the result in Lucene.
100 secs?
And you are calling 137MB a very big size?
Something doesn't feel right.
Are you doing two searches (1+1) or a lot of them (1+N)?
I really reread your mail a few times to understand if I had misread
seconds for milliseconds or MB for GB...
--
Roberto Ragusa mail at robertoragusa.it
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by neeraj shah <ne...@gmail.com>.
i have already tried this. I am fetching the info of "ABC" and then passing
then again in searcher with new Query then adding the required result.
but this method taking very very long time (more than 100 secs). my data
file is very big size would be around 137MB.
So was asking is there another way to get the result in Lucene.
On Sat, Jun 22, 2013 at 2:57 PM, Roberto Ragusa <ma...@robertoragusa.it>wrote:
> On 06/22/2013 08:07 AM, neeraj shah wrote:
> > Hello,
> >
> > Im using Lucene 3.6.2. and my file, which i indexed ,is something like
> > this :
> >
> > FIELD-1 FIELD-2 FIELD-3 FIELD-4
> > DOC1: A 1111 2222 ABC
> > DOC2: B 1111 KKKK ABC
> > DOC3: C 1111 JJJJ ABC
> > DOC4: D 1111 LLLL ABC
> > DOC5: E 1111 WWW ABC
> > DOC6: F 2222 CCCC DEF
> > DOC7: G 2222 SSSS DEF
> > DOC8: H 2222 FFFF DEF
> > DOC9: I 2222 QQQQ DEF
> >
> > now if i search over field "FIELD-3" and search for "222" ,i would get
> the
> > single result of "ABC".
> > My requirement is i want to fetch all results which contains "ABC". while
> > quering Field-4 value is not knwon so i can not put direct "ABC" for
> search.
> >
> > how can i do this?
>
> Two searches?
> Search 2222, identify doc1, get the stored field4 "ABC", now search
> for "field4:ABC".
> If the first search returns many results, I suppose you may want to add
> them all
> to the second search.
>
> --
> Roberto Ragusa mail at robertoragusa.it
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
With Regards,
Neeraj Kumar Shah
+919819474146
Re: how to fetch qyery feild + other feild related results in Lucene3.6
Posted by Roberto Ragusa <ma...@robertoragusa.it>.
On 06/22/2013 08:07 AM, neeraj shah wrote:
> Hello,
>
> Im using Lucene 3.6.2. and my file, which i indexed ,is something like
> this :
>
> FIELD-1 FIELD-2 FIELD-3 FIELD-4
> DOC1: A 1111 2222 ABC
> DOC2: B 1111 KKKK ABC
> DOC3: C 1111 JJJJ ABC
> DOC4: D 1111 LLLL ABC
> DOC5: E 1111 WWW ABC
> DOC6: F 2222 CCCC DEF
> DOC7: G 2222 SSSS DEF
> DOC8: H 2222 FFFF DEF
> DOC9: I 2222 QQQQ DEF
>
> now if i search over field "FIELD-3" and search for "222" ,i would get the
> single result of "ABC".
> My requirement is i want to fetch all results which contains "ABC". while
> quering Field-4 value is not knwon so i can not put direct "ABC" for search.
>
> how can i do this?
Two searches?
Search 2222, identify doc1, get the stored field4 "ABC", now search
for "field4:ABC".
If the first search returns many results, I suppose you may want to add them all
to the second search.
--
Roberto Ragusa mail at robertoragusa.it
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org