You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ramya <ra...@sifycorp.com> on 2005/05/12 16:33:12 UTC
indexdir/segments (No such file or directory) lock file present..
1. I am trying to pump in large number of documents( to the tune of
50000) ... I use muliple threads and i depend on the internal locks of
lucene to synchronize the write access to the index.
try
{
index = new IndexWriter(d, new StandardAnalyzer(), false);
}
catch(IOException e)
{
index = new IndexWriter("indexdir", new
StandardAnalyzer(), true);
}
index.addDocument(x.doc);
index.close();
<then i add a docu
I have the following problem.
1. Somehow one lock file remains in the tmp directory. When this
happens i get the error
FileNotFoundException: /root/mails/mailindex/indexdir/segments (No such
file or directory)
2. I thought when the index is closed the lock is released internally.
If it had been the case y would the lock file remain in the tm dir??
2. The size of the index file is too large(Almost 50% of the original
document). I use the standard analyzer. Will using a different analyzer
help. What would i loose if i use a different one.
I use the version lucene-1.4.3
Any hellp would be appreciated.
--
Thanks and regards
Ramya Krishnan
********** DISCLAIMER **********
Information contained and transmitted by this E-MAIL is proprietary to
Sify Limited and is intended for use only by the individual or entity to
which it is addressed, and may contain information that is privileged,
confidential or exempt from disclosure under applicable law. If this is a
forwarded message, the content of this E-MAIL may not have been sent with
the authority of the Company. If you are not the intended recipient, an
agent of the intended recipient or a person responsible for delivering the
information to the named recipient, you are notified that any use,
distribution, transmission, printing, copying or dissemination of this
information in any way or in any manner is strictly prohibited. If you have
received this communication in error, please delete this mail & notify us
immediately at admin@sifycorp.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: indexdir/segments (No such file or directory) lock file present..
Posted by Ramya <ra...@sifycorp.com>.
I have chenged my logic as suggested..
boolean create = !(new File("indexdir/segments").exists());
System.out.println("Number of objects writing to index=" + count);
index = new IndexWriter(d, new StandardAnalyzer(), create);
But still one problem persists..When i do a huge number of documents,
the files except the segment files get removed..
and the segment file is just 8K
1. How does the locking work.. When another process/thread is writing
the index, will the current process block or exit with an exception.
This information is crucial because there might be number of users
updating the same index.
2. Wat would happen when a document has variable number of fields each
time i update the index
Example
First doc
Heading Part1
Heading Part1 Part2
Heading Part1 Part2 Part3
Could this be the reason why i loose my index data when i try to update
it???
--
Thanks and regards
Ramya Krishnan
********** DISCLAIMER **********
Information contained and transmitted by this E-MAIL is proprietary to
Sify Limited and is intended for use only by the individual or entity to
which it is addressed, and may contain information that is privileged,
confidential or exempt from disclosure under applicable law. If this is a
forwarded message, the content of this E-MAIL may not have been sent with
the authority of the Company. If you are not the intended recipient, an
agent of the intended recipient or a person responsible for delivering the
information to the named recipient, you are notified that any use,
distribution, transmission, printing, copying or dissemination of this
information in any way or in any manner is strictly prohibited. If you have
received this communication in error, please delete this mail & notify us
immediately at admin@sifycorp.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: indexdir/segments (No such file or directory) lock file present..
Posted by Monsur Hossain <mo...@monsur.com>.
Ramya. I don't have an answer to your specific lock file question, but
a couple thoughts.
You say you're using multiple threads to index 50,000 documents. Have
you tried a single thread version first? I'd try that, and then scale
out to multiple threads as needed. We index over ten times that many
documents, and the entire application finishes in under an hour (this
includes loading the data from a relational database. However this is
on a fast server, your results may vary).
Also, you're using a try/catch block to dictate programming logic, which
could be causing issues. Instead, try something like:
boolean create = !DirectoryExists(d);
index = new IndexWriter(d, new StandardAnalyzer(), create);
The DirectoryExists() function just indicates whether your index exists
(you could also check if the segments file exists).
Monsur
> -----Original Message-----
> From: Ramya [mailto:ramya_krishnan@sifycorp.com]
> Sent: Thursday, May 12, 2005 10:33 AM
> To: java-user@lucene.apache.org
> Subject: indexdir/segments (No such file or directory) lock
> file present..
>
>
> 1. I am trying to pump in large number of documents( to the tune of
> 50000) ... I use muliple threads and i depend on the internal
> locks of
> lucene to synchronize the write access to the index.
>
> try
> {
> index = new IndexWriter(d, new
> StandardAnalyzer(), false);
> }
> catch(IOException e)
> {
> index = new IndexWriter("indexdir", new
> StandardAnalyzer(), true);
> }
> index.addDocument(x.doc);
> index.close();
>
> <then i add a docu
>
> I have the following problem.
> 1. Somehow one lock file remains in the tmp directory. When this
> happens i get the error
> FileNotFoundException:
> /root/mails/mailindex/indexdir/segments (No such
> file or directory)
> 2. I thought when the index is closed the lock is released
> internally.
> If it had been the case y would the lock file remain in the tm dir??
>
> 2. The size of the index file is too large(Almost 50% of the original
> document). I use the standard analyzer. Will using a
> different analyzer
> help. What would i loose if i use a different one.
>
>
>
> I use the version lucene-1.4.3
>
>
> Any hellp would be appreciated.
>
>
>
>
> --
> Thanks and regards
> Ramya Krishnan
>
> ********** DISCLAIMER **********
> Information contained and transmitted by this E-MAIL is
> proprietary to
> Sify Limited and is intended for use only by the individual
> or entity to
> which it is addressed, and may contain information that is
> privileged,
> confidential or exempt from disclosure under applicable law.
> If this is a
> forwarded message, the content of this E-MAIL may not have
> been sent with
> the authority of the Company. If you are not the intended
> recipient, an
> agent of the intended recipient or a person responsible for
> delivering the
> information to the named recipient, you are notified that any use,
> distribution, transmission, printing, copying or
> dissemination of this
> information in any way or in any manner is strictly
> prohibited. If you have
> received this communication in error, please delete this mail
> & notify us
> immediately at admin@sifycorp.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org