You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by mchaput <mc...@aw.sgi.com> on 2003/01/17 21:31:07 UTC
Index dir lock issue
Late last year there was some discussion on the user list regarding
Lucene's attempt to lock the index before a search failing because of
permissions. Doug suggested one good solution (a flag to not try to
lock) and sombody else suggested a great one (check write permissions,
and if you can't write, don't bother locking because it doesn't matter
anyway).
I was wondering if anything ever came of that discussion, because I'm
really running up against this right now. We'd really prefer not to
install something and have to make the index dir world writable.
I suppose it would be too much to hope that patches are available? ;)
Barring that, can a Lucene guru suggest how much work it would be for a
Lucene newbie (ie me) to make this change in the source, and where I
would begin?
Thanks very much!
Matt
--
|
Matt Chaput | A l i a s | W a v e f r o n t
Information Designer | 210 King St. E. Toronto, ON, Canada M5A 1J7
mchaput@aw.sgi.com | (416) 874-8268
|
"A goddamned ray of sunshine all the goddamned time" --Sparkle Hayter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Searcher is not returning the records - help please
Posted by Michael Wechner <mi...@wyona.org>.
Are you using HTMLDocument for indexing? The name of the content field is
"contents" and not "content" which you are using within your code (plz
see below).
Vinu SB wrote:
>Hi,
>I am relatively new to Lucene. My indexing process is
>going fine, as and when I upload the files. But when I
>search the index file for those existing words, the
>'hits' return no records. Can somebody point out what
>could be my mistake in the following code?
>
>Any help/suggestions are appreciated.
>Thanks,
>Vinu
>----------------------------------------------------
>package com.proj.search;
>
>import java.io.File;
>import java.io.IOException;
>import java.util.ArrayList;
>import com.proj.Repository.DocumentForm;
>import org.apache.struts.action.ActionError;
>import org.apache.struts.action.ActionErrors;
>
>import org.apache.lucene.analysis.*;
>import org.apache.lucene.document.Document;
>import org.apache.lucene.queryParser.QueryParser;
>import org.apache.lucene.queryParser.ParseException;
>import org.apache.lucene.index.IndexReader;
>import org.apache.lucene.search.IndexSearcher;
>import org.apache.lucene.search.Query;
>import org.apache.lucene.search.Hits;
>
>public class SearchIndexDir {
>
> public static ArrayList SearchFiles(String indexDir,
>String docDir, String queryString, int BUCKET,
>ActionErrors errors)
> {
> IndexSearcher searcher = null;
> Query query = null;
> Hits hits = null;
> ArrayList list = null;
>
> try {
> searcher = new
>IndexSearcher(IndexReader.open(indexDir));
> // searcher = new IndexSearcher(indexDir);
> } catch (Exception e) {
> errors.add("missingIndex", new
>ActionError("errors.search.lucene.index.missing"));
>
> }
>
> if (errors.empty())
> {
> if ( queryString.equals("") || null == queryString)
> {
> errors.add("emptyQueryString", new
>ActionError("errors.search.lucene.enter.query.missing"));
>
> }
> Analyzer analyzer = new StopAnalyzer();
> try { // search the entire content
> query = QueryParser.parse(queryString,
>"content", analyzer);
>
Try using "contents" instead of "content"
HTH
Michael
> // System.out.println(query.toString());
> } catch (ParseException e) {
> // close(searcher);
> errors.add("noResults", new
>ActionError("errors.search.lucene.results.notfound"));
> }
> }
>
>
> // iterate through the results, which is an array of
>documents
> if (errors.empty() && searcher != null) {
> int start = 0;
>
>
> try
> {
> hits = searcher.search(query);
> }
> catch (IOException e)
> {
> System.err.println("search "+e);
> }
>
> int end = Math.min(BUCKET, hits.length());
>
> if(hits.length() > 0)
> {
> list = new ArrayList();
> Document doc = null;
> int length = docDir.length()+1; // plus the '/'
>
> for(int i = start; i < end; i++)
> {
> try
> {
> doc = hits.doc(i); //get the next
>document
> }
> catch (IOException e)
> {
> System.out.println("get doc from hits "+e);
> }
> //get its url
> String url = doc.get("url");
> url = url.substring(length);
> int _ind = url.indexOf("_");
> String id = url.substring(0, _ind);
> System.out.println("Found "+id);
> DocumentForm form = new
>DocumentForm(id);
> list.add(form);
>
> }
> }
> }
>
> return list;
> }
> }
>
>__________________________________________________
>Do you Yahoo!?
>Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
>http://mailplus.yahoo.com
>
>--
>To unsubscribe, e-mail: <ma...@jakarta.apache.org>
>For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Searcher is not returning the records - help please
Posted by Vinu SB <vi...@yahoo.com>.
Hi,
I am relatively new to Lucene. My indexing process is
going fine, as and when I upload the files. But when I
search the index file for those existing words, the
'hits' return no records. Can somebody point out what
could be my mistake in the following code?
Any help/suggestions are appreciated.
Thanks,
Vinu
----------------------------------------------------
package com.proj.search;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import com.proj.Repository.DocumentForm;
import org.apache.struts.action.ActionError;
import org.apache.struts.action.ActionErrors;
import org.apache.lucene.analysis.*;
import org.apache.lucene.document.Document;
import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.queryParser.ParseException;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.Hits;
public class SearchIndexDir {
public static ArrayList SearchFiles(String indexDir,
String docDir, String queryString, int BUCKET,
ActionErrors errors)
{
IndexSearcher searcher = null;
Query query = null;
Hits hits = null;
ArrayList list = null;
try {
searcher = new
IndexSearcher(IndexReader.open(indexDir));
// searcher = new IndexSearcher(indexDir);
} catch (Exception e) {
errors.add("missingIndex", new
ActionError("errors.search.lucene.index.missing"));
}
if (errors.empty())
{
if ( queryString.equals("") || null == queryString)
{
errors.add("emptyQueryString", new
ActionError("errors.search.lucene.enter.query.missing"));
}
Analyzer analyzer = new StopAnalyzer();
try { // search the entire content
query = QueryParser.parse(queryString,
"content", analyzer);
// System.out.println(query.toString());
} catch (ParseException e) {
// close(searcher);
errors.add("noResults", new
ActionError("errors.search.lucene.results.notfound"));
}
}
// iterate through the results, which is an array of
documents
if (errors.empty() && searcher != null) {
int start = 0;
try
{
hits = searcher.search(query);
}
catch (IOException e)
{
System.err.println("search "+e);
}
int end = Math.min(BUCKET, hits.length());
if(hits.length() > 0)
{
list = new ArrayList();
Document doc = null;
int length = docDir.length()+1; // plus the '/'
for(int i = start; i < end; i++)
{
try
{
doc = hits.doc(i); //get the next
document
}
catch (IOException e)
{
System.out.println("get doc from hits "+e);
}
//get its url
String url = doc.get("url");
url = url.substring(length);
int _ind = url.indexOf("_");
String id = url.substring(0, _ind);
System.out.println("Found "+id);
DocumentForm form = new
DocumentForm(id);
list.add(form);
}
}
}
return list;
}
}
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: NewBe: help on this code
Posted by Otis Gospodnetic <ot...@yahoo.com>.
First thing you may want to change are those empty catch blocks.
If this is running as your web app user, it looks like it is a, make
sure that this user can write to 'indexFIle' (better name would be
indexDir).
Otis
--- Vinu SB <vi...@yahoo.com> wrote:
> Hi,
> I am relativly new to Lucene and I am trying to index
> the docs, as and when they are uploaded to the system,
> by calling the following code: I keep getting 'segment
> file not found error (thru debugger), when I was
> expecting it to create those index files.
>
> Can anybody look into the following code and tell me
> what is my mistake/errors? Any help on this is
> appreciated. Thanks,
> Vinu
> ---------------------------------------------------
> Sorry for the previos message with wrong subject line
> ----------------------------------------------------
> package com.proj.search;
> import java.io.*;
>
> import org.apache.struts.action.ActionErrors;
> import org.apache.struts.action.ActionError;
>
> import org.apache.lucene.index.IndexWriter;
> import org.apache.lucene.analysis.standard.*;
> import org.apache.lucene.document.*;
>
> public class FileIndexer {
> public static void indexDoc( File file, String
> indexFile, ActionErrors errors) {
>
> IndexWriter writer = null;
> try {
> File f;
> boolean create = true;
> // create index if the directory does not exist
> if ((f = new File(indexFile)).exists() &&
> f.isDirectory()) {
> create = false;
> } else {
> create = true;
> }
>
> writer = new IndexWriter(indexFile, new
> StandardAnalyzer(), create);
> writer.mergeFactor = 20;
> // now add this document to the Indexing proces:
>
> writer.addDocument(Document(file));
> writer.optimize();
> } catch (Exception e) {
> errors.add("unableToIndex", new
> ActionError("errors.search.lucene.index.unable"));
>
>
> } finally {
> close(writer);
> }
> }
> // close writer
> public static void close(IndexWriter writer) {
> if(null != writer) {
> try {
> writer.close();
> } catch(Exception e) {
> }
> }
> }
>
> // Lucene can only index objects of type Document.
> public static Document Document(File file) {
>
> FileInputStream fis=null;
> Reader reader = null;
> Document doc = null;
> if (null !=file) {
> try{
> fis=new FileInputStream(file);
> reader = new BufferedReader(new
> InputStreamReader(fis));
> doc = new Document();
> doc.add(Field.Text("content", reader));
> } catch (Exception e) {
> }
> }
> return doc;
> }
>
> }
>
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
NewBe: help on this code
Posted by Vinu SB <vi...@yahoo.com>.
Hi,
I am relativly new to Lucene and I am trying to index
the docs, as and when they are uploaded to the system,
by calling the following code: I keep getting 'segment
file not found error (thru debugger), when I was
expecting it to create those index files.
Can anybody look into the following code and tell me
what is my mistake/errors? Any help on this is
appreciated. Thanks,
Vinu
---------------------------------------------------
Sorry for the previos message with wrong subject line
----------------------------------------------------
package com.proj.search;
import java.io.*;
import org.apache.struts.action.ActionErrors;
import org.apache.struts.action.ActionError;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.analysis.standard.*;
import org.apache.lucene.document.*;
public class FileIndexer {
public static void indexDoc( File file, String
indexFile, ActionErrors errors) {
IndexWriter writer = null;
try {
File f;
boolean create = true;
// create index if the directory does not exist
if ((f = new File(indexFile)).exists() &&
f.isDirectory()) {
create = false;
} else {
create = true;
}
writer = new IndexWriter(indexFile, new
StandardAnalyzer(), create);
writer.mergeFactor = 20;
// now add this document to the Indexing proces:
writer.addDocument(Document(file));
writer.optimize();
} catch (Exception e) {
errors.add("unableToIndex", new
ActionError("errors.search.lucene.index.unable"));
} finally {
close(writer);
}
}
// close writer
public static void close(IndexWriter writer) {
if(null != writer) {
try {
writer.close();
} catch(Exception e) {
}
}
}
// Lucene can only index objects of type Document.
public static Document Document(File file) {
FileInputStream fis=null;
Reader reader = null;
Document doc = null;
if (null !=file) {
try{
fis=new FileInputStream(file);
reader = new BufferedReader(new
InputStreamReader(fis));
doc = new Document();
doc.add(Field.Text("content", reader));
} catch (Exception e) {
}
}
return doc;
}
}
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Index dir lock issue
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Stealing somebody else's thread? Bad manners!
Otis
--- Vinu SB <vi...@yahoo.com> wrote:
> Hi,
> I am relativly new to Lucene and I am trying to index
> the docs, as and when they are uploaded to the system,
> by calling the following code: I keep getting 'segment
> file not found error (thru debugger), when I was
> expecting it to create those index files.
>
> Can anybody look into the following code and tell me
> what is my mistake/errors? Any help on this is
> appreciated. Thanks,
> Vinu
> -------------
> package com.proj.search;
> import java.io.*;
>
> import org.apache.struts.action.ActionErrors;
> import org.apache.struts.action.ActionError;
>
> import org.apache.lucene.index.IndexWriter;
> import org.apache.lucene.analysis.standard.*;
> import org.apache.lucene.document.*;
>
> public class FileIndexer {
> public static void indexDoc( File file, String
> indexFile, ActionErrors errors) {
>
> IndexWriter writer = null;
> try {
> File f;
> boolean create = true;
> // create index if the directory does not exist
> if ((f = new File(indexFile)).exists() &&
> f.isDirectory()) {
> create = false;
> } else {
> create = true;
> }
>
> writer = new IndexWriter(indexFile, new
> StandardAnalyzer(), create);
> writer.mergeFactor = 20;
> // now add this document to the Indexing proces:
>
> writer.addDocument(Document(file));
> writer.optimize();
> } catch (Exception e) {
> errors.add("unableToIndex", new
> ActionError("errors.search.lucene.index.unable"));
>
>
> } finally {
> close(writer);
> }
> }
> // close writer
> public static void close(IndexWriter writer) {
> if(null != writer) {
> try {
> writer.close();
> } catch(Exception e) {
> }
> }
> }
>
> // Lucene can only index objects of type Document.
> public static Document Document(File file) {
>
> FileInputStream fis=null;
> Reader reader = null;
> Document doc = null;
> if (null !=file) {
> try{
> fis=new FileInputStream(file);
> reader = new BufferedReader(new
> InputStreamReader(fis));
> doc = new Document();
> doc.add(Field.Text("content", reader));
> } catch (Exception e) {
> }
> }
> return doc;
> }
>
> }
>
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Index dir lock issue
Posted by Vinu SB <vi...@yahoo.com>.
Hi,
I am relativly new to Lucene and I am trying to index
the docs, as and when they are uploaded to the system,
by calling the following code: I keep getting 'segment
file not found error (thru debugger), when I was
expecting it to create those index files.
Can anybody look into the following code and tell me
what is my mistake/errors? Any help on this is
appreciated. Thanks,
Vinu
-------------
package com.proj.search;
import java.io.*;
import org.apache.struts.action.ActionErrors;
import org.apache.struts.action.ActionError;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.analysis.standard.*;
import org.apache.lucene.document.*;
public class FileIndexer {
public static void indexDoc( File file, String
indexFile, ActionErrors errors) {
IndexWriter writer = null;
try {
File f;
boolean create = true;
// create index if the directory does not exist
if ((f = new File(indexFile)).exists() &&
f.isDirectory()) {
create = false;
} else {
create = true;
}
writer = new IndexWriter(indexFile, new
StandardAnalyzer(), create);
writer.mergeFactor = 20;
// now add this document to the Indexing proces:
writer.addDocument(Document(file));
writer.optimize();
} catch (Exception e) {
errors.add("unableToIndex", new
ActionError("errors.search.lucene.index.unable"));
} finally {
close(writer);
}
}
// close writer
public static void close(IndexWriter writer) {
if(null != writer) {
try {
writer.close();
} catch(Exception e) {
}
}
}
// Lucene can only index objects of type Document.
public static Document Document(File file) {
FileInputStream fis=null;
Reader reader = null;
Document doc = null;
if (null !=file) {
try{
fis=new FileInputStream(file);
reader = new BufferedReader(new
InputStreamReader(fis));
doc = new Document();
doc.add(Field.Text("content", reader));
} catch (Exception e) {
}
}
return doc;
}
}
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Index dir lock issue
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This question is more appropriate for lucene-user.
You may also get more help there.
While locking is disabled with a global system property, you normally
set properties with -D command line options when invoking java. If you
do want locks, don't use -D.
I am assuming that you are not mixing indices that need locking and
those that should not be locked in the same invocation of your Java
application.
There is also nothing in the code that disables locks forever - only
during the Java invocation with -DdisableLuceneLocks=true.
If you subsequently start your application without that -D.... business
the locks will be used again.
Otis
--- mchaput <mc...@aw.sgi.com> wrote:
> Otis Gospodnetic wrote:
> > Look at the CHANGES.txt file, some changes dealing with lock files
> were
> > made a while ago. Have you tried using that?
>
> I checked it out, but it's seems to be a global system property-- as
> I
> understand it, it would be only be useful if all indexes on any given
>
> machine are only and forever going to be read-only. I'd prefer a more
>
> flexible approach that expects write access but doesn't break
> searching
> if it doesn't get it.
>
> Right now suggestions from other people around here are running
> toward
> modifying the code to write lock files to /tmp or C:\temp, however
> I'm
> pretty sure that would be a bit of a serious change for me to do
> given
> that I'm not familiar with the source.
>
> Any thoughts on a nice, easy way to get around this?
>
> Thanks for listening!
>
> Matt
>
>
> --
> |
> Matt Chaput | A l i a s | W a v e f r o n t
> Information Designer | 210 King St. E. Toronto, ON, Canada M5A 1J7
> mchaput@aw.sgi.com | (416) 874-8268
> |
> "A goddamned ray of sunshine all the goddamned time" --Sparkle Hayter
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Index dir lock issue
Posted by mchaput <mc...@aw.sgi.com>.
Otis Gospodnetic wrote:
> Look at the CHANGES.txt file, some changes dealing with lock files were
> made a while ago. Have you tried using that?
I checked it out, but it's seems to be a global system property-- as I
understand it, it would be only be useful if all indexes on any given
machine are only and forever going to be read-only. I'd prefer a more
flexible approach that expects write access but doesn't break searching
if it doesn't get it.
Right now suggestions from other people around here are running toward
modifying the code to write lock files to /tmp or C:\temp, however I'm
pretty sure that would be a bit of a serious change for me to do given
that I'm not familiar with the source.
Any thoughts on a nice, easy way to get around this?
Thanks for listening!
Matt
--
|
Matt Chaput | A l i a s | W a v e f r o n t
Information Designer | 210 King St. E. Toronto, ON, Canada M5A 1J7
mchaput@aw.sgi.com | (416) 874-8268
|
"A goddamned ray of sunshine all the goddamned time" --Sparkle Hayter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Index dir lock issue
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Look at the CHANGES.txt file, some changes dealing with lock files were
made a while ago. Have you tried using that?
Otis
--- mchaput <mc...@aw.sgi.com> wrote:
> Late last year there was some discussion on the user list regarding
> Lucene's attempt to lock the index before a search failing because of
>
> permissions. Doug suggested one good solution (a flag to not try to
> lock) and sombody else suggested a great one (check write
> permissions,
> and if you can't write, don't bother locking because it doesn't
> matter
> anyway).
>
> I was wondering if anything ever came of that discussion, because I'm
>
> really running up against this right now. We'd really prefer not to
> install something and have to make the index dir world writable.
>
> I suppose it would be too much to hope that patches are available? ;)
>
> Barring that, can a Lucene guru suggest how much work it would be for
> a
> Lucene newbie (ie me) to make this change in the source, and where I
> would begin?
>
> Thanks very much!
>
> Matt
>
>
> --
> |
> Matt Chaput | A l i a s | W a v e f r o n t
> Information Designer | 210 King St. E. Toronto, ON, Canada M5A 1J7
> mchaput@aw.sgi.com | (416) 874-8268
> |
> "A goddamned ray of sunshine all the goddamned time" --Sparkle Hayter
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>