You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Floyd Wu <fl...@gmail.com> on 2009/04/23 08:23:45 UTC
Luke-0.9.2 cannot open index files
Hi does anyone have the some problem?
My index files is built by Lucene.Net 2.3.1 and it can't be opend by java
tool "Luke-0.9.2".
I tried to open the same index files with Luke-0.8.1 and it works fine.
Attachment is my index files.
Every time I opened index files with Luke-0.9.2, Luke will show a message
"Read past EOF".
Please help on this.
Re: Luke-0.9.2 cannot open index files
Posted by Floyd Wu <fl...@gmail.com>.
Hi Digy,
writer is LuceneIndexWriter
actually we do a wrapper to LuceneIndexWriter coz we plant to change index
writer for future implementation.
Why using lock(), yes we are in multi-thread environment.
2009/4/23 Digy <di...@gmail.com>
> I don't see anything in your code related with Lucene.Net. What is
> "writer"?
> What does "AddContent" do to update the index? Why do use "lock", is your
> code multi-threaded? etc.
>
> If it is a bug of Lucene.Net, it is not easy to find it with the info you
> supplied.
>
> DIGY
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 1:26 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
> Hi Digy,
>
> here is my code for add a document to lucene
>
> internal static void AddContent(Content content)
> {
> lock (writer)
> {
> writer.Open();
> try
> {
> writer.AddContent(content);
> }
> finally
> {
> writer.Close();
> }
> }
> }
>
> No magic, just like ordinary usage.
>
> Floyd
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
> > Hi Floyd,
> >
> > Is it possible to send your indexing code?
> >
> > DIGY
> >
> > -----Original Message-----
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: Thursday, April 23, 2009 12:56 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: Luke-0.9.2 cannot open index files
> >
> > Hello Digy,
> >
> > Actually my index files that you owned is created by Lucene.Net-2.3.2 in
> > trunk this morning.
> >
> > Floyd
> >
> >
> >
> > 2009/4/23 Digy <di...@gmail.com>
> >
> > > Hi Floyd,
> > > I am using Luke 0.9.1. Altought it has the same problem while opening
> > your
> > > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
> > > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
> > >
> > > DIGY
> > >
> > > -----Original Message-----
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > > Sent: Thursday, April 23, 2009 10:49 AM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Re: Luke-0.9.2 cannot open index files
> > >
> > > Hi Jeroen,
> > > First I'm sure that close the writer after doing my job. And if not
> > > properly
> > > closed writer is the main point, why Luke-0.8.1 could open? So I guess
> > > there
> > > must something wrong or changed between Lucene.Net or Luke.
> > >
> > > I've asked the same question to the author of Luke, and he( or she)
> told
> > me
> > > that index files format mabe changed but not very sure.
> > >
> > >
> > >
> > > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
> > >
> > > > Hi,
> > > >
> > > > Just a thought:
> > > > Are you sure your IndexWriter was properly closed?
> > > > I have had the same symptoms when I forgot to call the Close()
> method.
> > > >
> > > > Jeroen
> > > >
> > > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > > > Sent: donderdag 23 april 2009 8:24
> > > > To: lucene-net-user@incubator.apache.org
> > > > Subject: Luke-0.9.2 cannot open index files
> > > >
> > > > Hi does anyone have the some problem?
> > > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
> > java
> > > > tool "Luke-0.9.2".
> > > > I tried to open the same index files with Luke-0.8.1 and it works
> fine.
> > > >
> > > > Attachment is my index files.
> > > > Every time I opened index files with Luke-0.9.2, Luke will show a
> > message
> > > > "Read past EOF".
> > > >
> > > > Please help on this.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
RE: Luke-0.9.2 cannot open index files
Posted by Digy <di...@gmail.com>.
I don't see anything in your code related with Lucene.Net. What is "writer"?
What does "AddContent" do to update the index? Why do use "lock", is your
code multi-threaded? etc.
If it is a bug of Lucene.Net, it is not easy to find it with the info you
supplied.
DIGY
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Thursday, April 23, 2009 1:26 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Luke-0.9.2 cannot open index files
Hi Digy,
here is my code for add a document to lucene
internal static void AddContent(Content content)
{
lock (writer)
{
writer.Open();
try
{
writer.AddContent(content);
}
finally
{
writer.Close();
}
}
}
No magic, just like ordinary usage.
Floyd
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
>
> Is it possible to send your indexing code?
>
> DIGY
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 12:56 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
> Hello Digy,
>
> Actually my index files that you owned is created by Lucene.Net-2.3.2 in
> trunk this morning.
>
> Floyd
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
> > Hi Floyd,
> > I am using Luke 0.9.1. Altought it has the same problem while opening
> your
> > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
> > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
> >
> > DIGY
> >
> > -----Original Message-----
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: Thursday, April 23, 2009 10:49 AM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: Luke-0.9.2 cannot open index files
> >
> > Hi Jeroen,
> > First I'm sure that close the writer after doing my job. And if not
> > properly
> > closed writer is the main point, why Luke-0.8.1 could open? So I guess
> > there
> > must something wrong or changed between Lucene.Net or Luke.
> >
> > I've asked the same question to the author of Luke, and he( or she) told
> me
> > that index files format mabe changed but not very sure.
> >
> >
> >
> > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
> >
> > > Hi,
> > >
> > > Just a thought:
> > > Are you sure your IndexWriter was properly closed?
> > > I have had the same symptoms when I forgot to call the Close() method.
> > >
> > > Jeroen
> > >
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > > Sent: donderdag 23 april 2009 8:24
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Luke-0.9.2 cannot open index files
> > >
> > > Hi does anyone have the some problem?
> > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
> java
> > > tool "Luke-0.9.2".
> > > I tried to open the same index files with Luke-0.8.1 and it works
fine.
> > >
> > > Attachment is my index files.
> > > Every time I opened index files with Luke-0.9.2, Luke will show a
> message
> > > "Read past EOF".
> > >
> > > Please help on this.
> > >
> > >
> > >
> > >
> >
> >
>
>
RE: Luke-0.9.2 cannot open index files
Posted by "Granroth, Neal V." <ne...@thermofisher.com>.
Floyd,
I think Digy is probably correct, the index you posted earlier (documents.zip) appears corrupt.
I looked it over as I have been trying to collect the information needed to document a bug in Luke related to the inability to open indexes which have not been compacted after deleting documents.
Inspecting your index there is 1 live document, and 2 deleted.
The live one has a number of fields that do not look useful:
Field: ???, Value:
Field: ???, Value:
Field: ????, Value:
Field: ????, Value:
Field: ????, Value:
Field: ??????, Value:
Field: ??, Value:
Field: ??, Value:
Field: ??, Value:
If I run optimize to compact the index; Luke no longer displays an error, but it acts as if the index is entirely empty.
Inspecting the compacted index, there are now 3 live documents.
That the number of live documents changed as the result of optimize and that there are so many meaningless fields points to a corrupt index.
-- Neal
________________________________________
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Thursday, April 23, 2009 9:33 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Luke-0.9.2 cannot open index files
Hi DIGY
Okay thanks, but I found another strange thing that the same system but different index files can be opened by Luke -0.9.2.
What's the difference between two index file(one is sent before, the other as attachment of this mail) were fields count.
You could compare these two by using Luke-0.8.1.
Floyd
2009/4/24 Digy <di...@gmail.com>
Hi Floyd,
Don't take the example seriously. It is just to show, how indexing threads
can be blocked, and when the application exits you get an corrupted index.
DIGY.
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Thursday, April 23, 2009 6:48 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Luke-0.9.2 cannot open index files
Hi Digy
If the root of problem is lock(), I wonder why the same index files can be
opened with previous version of Luke? i'm not very sure what happened with
new version of Lucene.Net. But I will try to modified my code with no lock
statement.
FYI, in single user situation, the index files generated from my system
could not be opened with new Luke also.
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
>
>
>
> For ex, see the sample code below: If you lock IndexWriter the indexing
> code
> gets stuck. But If you don't use the lock and let the Lucene do it
> internally, indexing goes fine. Maybe your index gets corrupted with such
a
> code?
>
>
>
> DIGY.
>
>
>
>
>
> using System;
>
> using System.Collections.Generic;
>
> using System.ComponentModel;
>
> using System.Data;
>
> using System.Drawing;
>
> using System.Text;
>
> using System.Windows.Forms;
>
>
>
> using System.IO;
>
> using Lucene.Net.Analysis.Unicode;
>
> using Lucene.Net.Index;
>
> using Lucene.Net.Search;
>
> using Lucene.Net.QueryParsers;
>
> using Lucene.Net.Documents;
>
>
>
> using System.Threading;
>
>
>
>
>
>
>
>
>
> namespace LuceneTest
>
> {
>
> public partial class IndexingSetsStuck : Form
>
> {
>
> public IndexingSetsStuck()
>
> {
>
> InitializeComponent();
>
> }
>
>
>
> private void button1_Click(object sender, EventArgs e)
>
> {
>
> Thread t = new Thread(new ThreadStart(CreateIndex));
>
> t.IsBackground = true;
>
> t.Start();
>
> }
>
>
>
> private void timer1_Tick(object sender, EventArgs e)
>
> {
>
> this.Text = Count.ToString();
>
> }
>
>
>
>
>
> string INDEX = @"TestIndex";
>
> IndexWriter writer = null;
>
> object LockObject = new object();
>
> int Count = 0;
>
>
>
> void Worker()
>
> {
>
> Field f = new Field("Line", "", Field.Store.YES,
> Field.Index.TOKENIZED);
>
> Document d = new Document();
>
> d.Add(f);
>
>
>
> string line = "";
>
> for (int i = 0; i < 50000; i++)
>
> {
>
> line = i.ToString().PadLeft(12, '0') + " " +
> i.ToString("x").PadLeft(10, '0');
>
> f.SetValue(line);
>
> writer.AddDocument(d);
>
>
>
> lock (LockObject) Count++;
>
> }
>
> }
>
>
>
> void CreateIndex()
>
> {
>
> writer = new IndexWriter(INDEX, new
> Lucene.Net.Analysis.WhitespaceAnalyzer(), true);
>
>
>
> /*****!!! ?????? !!!*****/
>
> //lock (writer) //Indexing gets stuck when this line is
> uncommented
>
> {
>
> Thread[] threads = new Thread[5];
>
> for (int i = 0; i < threads.Length; i++)
>
> {
>
> threads[i] = new Thread(new ThreadStart(Worker));
>
> threads[i].IsBackground = true;
>
> threads[i].Start();
>
> }
>
> for (int i = 0; i < threads.Length; i++)
>
> {
>
> threads[i].Join();
>
> }
>
> }
>
> writer.Close(true);
>
>
>
> MessageBox.Show("Indexing Completed");
>
> }
>
> }
>
> }
>
>
>
>
>
>
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 1:26 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
>
>
> Hi Digy,
>
>
>
> here is my code for add a document to lucene
>
>
>
> internal static void AddContent(Content content)
>
> {
>
> lock (writer)
>
> {
>
> writer.Open();
>
> try
>
> {
>
> writer.AddContent(content);
>
> }
>
> finally
>
> {
>
> writer.Close();
>
> }
>
> }
>
> }
>
>
>
> No magic, just like ordinary usage.
>
>
>
> Floyd
>
>
>
>
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
>
>
> > Hi Floyd,
>
> >
>
> > Is it possible to send your indexing code?
>
> >
>
> > DIGY
>
> >
>
> > -----Original Message-----
>
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > Sent: Thursday, April 23, 2009 12:56 PM
>
> > To: lucene-net-user@incubator.apache.org
>
> > Subject: Re: Luke-0.9.2 cannot open index files
>
> >
>
> > Hello Digy,
>
> >
>
> > Actually my index files that you owned is created by Lucene.Net-2.3.2 in
>
> > trunk this morning.
>
> >
>
> > Floyd
>
> >
>
> >
>
> >
>
> > 2009/4/23 Digy <di...@gmail.com>
>
> >
>
> > > Hi Floyd,
>
> > > I am using Luke 0.9.1. Altought it has the same problem while opening
>
> > your
>
> > > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
>
> > > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
>
> > >
>
> > > DIGY
>
> > >
>
> > > -----Original Message-----
>
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > > Sent: Thursday, April 23, 2009 10:49 AM
>
> > > To: lucene-net-user@incubator.apache.org
>
> > > Subject: Re: Luke-0.9.2 cannot open index files
>
> > >
>
> > > Hi Jeroen,
>
> > > First I'm sure that close the writer after doing my job. And if not
>
> > > properly
>
> > > closed writer is the main point, why Luke-0.8.1 could open? So I guess
>
> > > there
>
> > > must something wrong or changed between Lucene.Net or Luke.
>
> > >
>
> > > I've asked the same question to the author of Luke, and he( or she)
> told
>
> > me
>
> > > that index files format mabe changed but not very sure.
>
> > >
>
> > >
>
> > >
>
> > > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
>
> > >
>
> > > > Hi,
>
> > > >
>
> > > > Just a thought:
>
> > > > Are you sure your IndexWriter was properly closed?
>
> > > > I have had the same symptoms when I forgot to call the Close()
> method.
>
> > > >
>
> > > > Jeroen
>
> > > >
>
> > > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > > > Sent: donderdag 23 april 2009 8:24
>
> > > > To: lucene-net-user@incubator.apache.org
>
> > > > Subject: Luke-0.9.2 cannot open index files
>
> > > >
>
> > > > Hi does anyone have the some problem?
>
> > > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
>
> > java
>
> > > > tool "Luke-0.9.2".
>
> > > > I tried to open the same index files with Luke-0.8.1 and it works
> fine.
>
> > > >
>
> > > > Attachment is my index files.
>
> > > > Every time I opened index files with Luke-0.9.2, Luke will show a
>
> > message
>
> > > > "Read past EOF".
>
> > > >
>
> > > > Please help on this.
>
> > > >
>
> > > >
>
> > > >
>
> > > >
>
> > >
>
> > >
>
> >
>
> >
>
>
Re: Luke-0.9.2 cannot open index files
Posted by Floyd Wu <fl...@gmail.com>.
Hi DIGY
Okay thanks, but I found another strange thing that the same system but
different index files can be opened by Luke -0.9.2.
What's the difference between two index file(one is sent before, the other
as attachment of this mail) were fields count.
You could compare these two by using Luke-0.8.1.
Floyd
2009/4/24 Digy <di...@gmail.com>
> Hi Floyd,
>
> Don't take the example seriously. It is just to show, how indexing threads
> can be blocked, and when the application exits you get an corrupted index.
>
> DIGY.
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 6:48 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
> Hi Digy
>
> If the root of problem is lock(), I wonder why the same index files can be
> opened with previous version of Luke? i'm not very sure what happened with
> new version of Lucene.Net. But I will try to modified my code with no lock
> statement.
> FYI, in single user situation, the index files generated from my system
> could not be opened with new Luke also.
>
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
> > Hi Floyd,
> >
> >
> >
> > For ex, see the sample code below: If you lock IndexWriter the indexing
> > code
> > gets stuck. But If you don't use the lock and let the Lucene do it
> > internally, indexing goes fine. Maybe your index gets corrupted with such
> a
> > code?
> >
> >
> >
> > DIGY.
> >
> >
> >
> >
> >
> > using System;
> >
> > using System.Collections.Generic;
> >
> > using System.ComponentModel;
> >
> > using System.Data;
> >
> > using System.Drawing;
> >
> > using System.Text;
> >
> > using System.Windows.Forms;
> >
> >
> >
> > using System.IO;
> >
> > using Lucene.Net.Analysis.Unicode;
> >
> > using Lucene.Net.Index;
> >
> > using Lucene.Net.Search;
> >
> > using Lucene.Net.QueryParsers;
> >
> > using Lucene.Net.Documents;
> >
> >
> >
> > using System.Threading;
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > namespace LuceneTest
> >
> > {
> >
> > public partial class IndexingSetsStuck : Form
> >
> > {
> >
> > public IndexingSetsStuck()
> >
> > {
> >
> > InitializeComponent();
> >
> > }
> >
> >
> >
> > private void button1_Click(object sender, EventArgs e)
> >
> > {
> >
> > Thread t = new Thread(new ThreadStart(CreateIndex));
> >
> > t.IsBackground = true;
> >
> > t.Start();
> >
> > }
> >
> >
> >
> > private void timer1_Tick(object sender, EventArgs e)
> >
> > {
> >
> > this.Text = Count.ToString();
> >
> > }
> >
> >
> >
> >
> >
> > string INDEX = @"TestIndex";
> >
> > IndexWriter writer = null;
> >
> > object LockObject = new object();
> >
> > int Count = 0;
> >
> >
> >
> > void Worker()
> >
> > {
> >
> > Field f = new Field("Line", "", Field.Store.YES,
> > Field.Index.TOKENIZED);
> >
> > Document d = new Document();
> >
> > d.Add(f);
> >
> >
> >
> > string line = "";
> >
> > for (int i = 0; i < 50000; i++)
> >
> > {
> >
> > line = i.ToString().PadLeft(12, '0') + " " +
> > i.ToString("x").PadLeft(10, '0');
> >
> > f.SetValue(line);
> >
> > writer.AddDocument(d);
> >
> >
> >
> > lock (LockObject) Count++;
> >
> > }
> >
> > }
> >
> >
> >
> > void CreateIndex()
> >
> > {
> >
> > writer = new IndexWriter(INDEX, new
> > Lucene.Net.Analysis.WhitespaceAnalyzer(), true);
> >
> >
> >
> > /*****!!! ?????? !!!*****/
> >
> > //lock (writer) //Indexing gets stuck when this line is
> > uncommented
> >
> > {
> >
> > Thread[] threads = new Thread[5];
> >
> > for (int i = 0; i < threads.Length; i++)
> >
> > {
> >
> > threads[i] = new Thread(new ThreadStart(Worker));
> >
> > threads[i].IsBackground = true;
> >
> > threads[i].Start();
> >
> > }
> >
> > for (int i = 0; i < threads.Length; i++)
> >
> > {
> >
> > threads[i].Join();
> >
> > }
> >
> > }
> >
> > writer.Close(true);
> >
> >
> >
> > MessageBox.Show("Indexing Completed");
> >
> > }
> >
> > }
> >
> > }
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: Thursday, April 23, 2009 1:26 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: Luke-0.9.2 cannot open index files
> >
> >
> >
> > Hi Digy,
> >
> >
> >
> > here is my code for add a document to lucene
> >
> >
> >
> > internal static void AddContent(Content content)
> >
> > {
> >
> > lock (writer)
> >
> > {
> >
> > writer.Open();
> >
> > try
> >
> > {
> >
> > writer.AddContent(content);
> >
> > }
> >
> > finally
> >
> > {
> >
> > writer.Close();
> >
> > }
> >
> > }
> >
> > }
> >
> >
> >
> > No magic, just like ordinary usage.
> >
> >
> >
> > Floyd
> >
> >
> >
> >
> >
> >
> >
> > 2009/4/23 Digy <di...@gmail.com>
> >
> >
> >
> > > Hi Floyd,
> >
> > >
> >
> > > Is it possible to send your indexing code?
> >
> > >
> >
> > > DIGY
> >
> > >
> >
> > > -----Original Message-----
> >
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> >
> > > Sent: Thursday, April 23, 2009 12:56 PM
> >
> > > To: lucene-net-user@incubator.apache.org
> >
> > > Subject: Re: Luke-0.9.2 cannot open index files
> >
> > >
> >
> > > Hello Digy,
> >
> > >
> >
> > > Actually my index files that you owned is created by Lucene.Net-2.3.2
> in
> >
> > > trunk this morning.
> >
> > >
> >
> > > Floyd
> >
> > >
> >
> > >
> >
> > >
> >
> > > 2009/4/23 Digy <di...@gmail.com>
> >
> > >
> >
> > > > Hi Floyd,
> >
> > > > I am using Luke 0.9.1. Altought it has the same problem while opening
> >
> > > your
> >
> > > > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
> >
> > > > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
> >
> > > >
> >
> > > > DIGY
> >
> > > >
> >
> > > > -----Original Message-----
> >
> > > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> >
> > > > Sent: Thursday, April 23, 2009 10:49 AM
> >
> > > > To: lucene-net-user@incubator.apache.org
> >
> > > > Subject: Re: Luke-0.9.2 cannot open index files
> >
> > > >
> >
> > > > Hi Jeroen,
> >
> > > > First I'm sure that close the writer after doing my job. And if not
> >
> > > > properly
> >
> > > > closed writer is the main point, why Luke-0.8.1 could open? So I
> guess
> >
> > > > there
> >
> > > > must something wrong or changed between Lucene.Net or Luke.
> >
> > > >
> >
> > > > I've asked the same question to the author of Luke, and he( or she)
> > told
> >
> > > me
> >
> > > > that index files format mabe changed but not very sure.
> >
> > > >
> >
> > > >
> >
> > > >
> >
> > > > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
> >
> > > >
> >
> > > > > Hi,
> >
> > > > >
> >
> > > > > Just a thought:
> >
> > > > > Are you sure your IndexWriter was properly closed?
> >
> > > > > I have had the same symptoms when I forgot to call the Close()
> > method.
> >
> > > > >
> >
> > > > > Jeroen
> >
> > > > >
> >
> > > > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> >
> > > > > Sent: donderdag 23 april 2009 8:24
> >
> > > > > To: lucene-net-user@incubator.apache.org
> >
> > > > > Subject: Luke-0.9.2 cannot open index files
> >
> > > > >
> >
> > > > > Hi does anyone have the some problem?
> >
> > > > > My index files is built by Lucene.Net 2.3.1 and it can't be opend
> by
> >
> > > java
> >
> > > > > tool "Luke-0.9.2".
> >
> > > > > I tried to open the same index files with Luke-0.8.1 and it works
> > fine.
> >
> > > > >
> >
> > > > > Attachment is my index files.
> >
> > > > > Every time I opened index files with Luke-0.9.2, Luke will show a
> >
> > > message
> >
> > > > > "Read past EOF".
> >
> > > > >
> >
> > > > > Please help on this.
> >
> > > > >
> >
> > > > >
> >
> > > > >
> >
> > > > >
> >
> > > >
> >
> > > >
> >
> > >
> >
> > >
> >
> >
>
>
RE: Luke-0.9.2 cannot open index files
Posted by Digy <di...@gmail.com>.
Hi Floyd,
Don't take the example seriously. It is just to show, how indexing threads
can be blocked, and when the application exits you get an corrupted index.
DIGY.
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Thursday, April 23, 2009 6:48 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Luke-0.9.2 cannot open index files
Hi Digy
If the root of problem is lock(), I wonder why the same index files can be
opened with previous version of Luke? i'm not very sure what happened with
new version of Lucene.Net. But I will try to modified my code with no lock
statement.
FYI, in single user situation, the index files generated from my system
could not be opened with new Luke also.
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
>
>
>
> For ex, see the sample code below: If you lock IndexWriter the indexing
> code
> gets stuck. But If you don't use the lock and let the Lucene do it
> internally, indexing goes fine. Maybe your index gets corrupted with such
a
> code?
>
>
>
> DIGY.
>
>
>
>
>
> using System;
>
> using System.Collections.Generic;
>
> using System.ComponentModel;
>
> using System.Data;
>
> using System.Drawing;
>
> using System.Text;
>
> using System.Windows.Forms;
>
>
>
> using System.IO;
>
> using Lucene.Net.Analysis.Unicode;
>
> using Lucene.Net.Index;
>
> using Lucene.Net.Search;
>
> using Lucene.Net.QueryParsers;
>
> using Lucene.Net.Documents;
>
>
>
> using System.Threading;
>
>
>
>
>
>
>
>
>
> namespace LuceneTest
>
> {
>
> public partial class IndexingSetsStuck : Form
>
> {
>
> public IndexingSetsStuck()
>
> {
>
> InitializeComponent();
>
> }
>
>
>
> private void button1_Click(object sender, EventArgs e)
>
> {
>
> Thread t = new Thread(new ThreadStart(CreateIndex));
>
> t.IsBackground = true;
>
> t.Start();
>
> }
>
>
>
> private void timer1_Tick(object sender, EventArgs e)
>
> {
>
> this.Text = Count.ToString();
>
> }
>
>
>
>
>
> string INDEX = @"TestIndex";
>
> IndexWriter writer = null;
>
> object LockObject = new object();
>
> int Count = 0;
>
>
>
> void Worker()
>
> {
>
> Field f = new Field("Line", "", Field.Store.YES,
> Field.Index.TOKENIZED);
>
> Document d = new Document();
>
> d.Add(f);
>
>
>
> string line = "";
>
> for (int i = 0; i < 50000; i++)
>
> {
>
> line = i.ToString().PadLeft(12, '0') + " " +
> i.ToString("x").PadLeft(10, '0');
>
> f.SetValue(line);
>
> writer.AddDocument(d);
>
>
>
> lock (LockObject) Count++;
>
> }
>
> }
>
>
>
> void CreateIndex()
>
> {
>
> writer = new IndexWriter(INDEX, new
> Lucene.Net.Analysis.WhitespaceAnalyzer(), true);
>
>
>
> /*****!!! ?????? !!!*****/
>
> //lock (writer) //Indexing gets stuck when this line is
> uncommented
>
> {
>
> Thread[] threads = new Thread[5];
>
> for (int i = 0; i < threads.Length; i++)
>
> {
>
> threads[i] = new Thread(new ThreadStart(Worker));
>
> threads[i].IsBackground = true;
>
> threads[i].Start();
>
> }
>
> for (int i = 0; i < threads.Length; i++)
>
> {
>
> threads[i].Join();
>
> }
>
> }
>
> writer.Close(true);
>
>
>
> MessageBox.Show("Indexing Completed");
>
> }
>
> }
>
> }
>
>
>
>
>
>
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 1:26 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
>
>
> Hi Digy,
>
>
>
> here is my code for add a document to lucene
>
>
>
> internal static void AddContent(Content content)
>
> {
>
> lock (writer)
>
> {
>
> writer.Open();
>
> try
>
> {
>
> writer.AddContent(content);
>
> }
>
> finally
>
> {
>
> writer.Close();
>
> }
>
> }
>
> }
>
>
>
> No magic, just like ordinary usage.
>
>
>
> Floyd
>
>
>
>
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
>
>
> > Hi Floyd,
>
> >
>
> > Is it possible to send your indexing code?
>
> >
>
> > DIGY
>
> >
>
> > -----Original Message-----
>
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > Sent: Thursday, April 23, 2009 12:56 PM
>
> > To: lucene-net-user@incubator.apache.org
>
> > Subject: Re: Luke-0.9.2 cannot open index files
>
> >
>
> > Hello Digy,
>
> >
>
> > Actually my index files that you owned is created by Lucene.Net-2.3.2 in
>
> > trunk this morning.
>
> >
>
> > Floyd
>
> >
>
> >
>
> >
>
> > 2009/4/23 Digy <di...@gmail.com>
>
> >
>
> > > Hi Floyd,
>
> > > I am using Luke 0.9.1. Altought it has the same problem while opening
>
> > your
>
> > > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
>
> > > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
>
> > >
>
> > > DIGY
>
> > >
>
> > > -----Original Message-----
>
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > > Sent: Thursday, April 23, 2009 10:49 AM
>
> > > To: lucene-net-user@incubator.apache.org
>
> > > Subject: Re: Luke-0.9.2 cannot open index files
>
> > >
>
> > > Hi Jeroen,
>
> > > First I'm sure that close the writer after doing my job. And if not
>
> > > properly
>
> > > closed writer is the main point, why Luke-0.8.1 could open? So I guess
>
> > > there
>
> > > must something wrong or changed between Lucene.Net or Luke.
>
> > >
>
> > > I've asked the same question to the author of Luke, and he( or she)
> told
>
> > me
>
> > > that index files format mabe changed but not very sure.
>
> > >
>
> > >
>
> > >
>
> > > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
>
> > >
>
> > > > Hi,
>
> > > >
>
> > > > Just a thought:
>
> > > > Are you sure your IndexWriter was properly closed?
>
> > > > I have had the same symptoms when I forgot to call the Close()
> method.
>
> > > >
>
> > > > Jeroen
>
> > > >
>
> > > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > > > Sent: donderdag 23 april 2009 8:24
>
> > > > To: lucene-net-user@incubator.apache.org
>
> > > > Subject: Luke-0.9.2 cannot open index files
>
> > > >
>
> > > > Hi does anyone have the some problem?
>
> > > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
>
> > java
>
> > > > tool "Luke-0.9.2".
>
> > > > I tried to open the same index files with Luke-0.8.1 and it works
> fine.
>
> > > >
>
> > > > Attachment is my index files.
>
> > > > Every time I opened index files with Luke-0.9.2, Luke will show a
>
> > message
>
> > > > "Read past EOF".
>
> > > >
>
> > > > Please help on this.
>
> > > >
>
> > > >
>
> > > >
>
> > > >
>
> > >
>
> > >
>
> >
>
> >
>
>
Re: Luke-0.9.2 cannot open index files
Posted by Floyd Wu <fl...@gmail.com>.
Hi Digy
If the root of problem is lock(), I wonder why the same index files can be
opened with previous version of Luke? i'm not very sure what happened with
new version of Lucene.Net. But I will try to modified my code with no lock
statement.
FYI, in single user situation, the index files generated from my system
could not be opened with new Luke also.
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
>
>
>
> For ex, see the sample code below: If you lock IndexWriter the indexing
> code
> gets stuck. But If you don't use the lock and let the Lucene do it
> internally, indexing goes fine. Maybe your index gets corrupted with such a
> code?
>
>
>
> DIGY.
>
>
>
>
>
> using System;
>
> using System.Collections.Generic;
>
> using System.ComponentModel;
>
> using System.Data;
>
> using System.Drawing;
>
> using System.Text;
>
> using System.Windows.Forms;
>
>
>
> using System.IO;
>
> using Lucene.Net.Analysis.Unicode;
>
> using Lucene.Net.Index;
>
> using Lucene.Net.Search;
>
> using Lucene.Net.QueryParsers;
>
> using Lucene.Net.Documents;
>
>
>
> using System.Threading;
>
>
>
>
>
>
>
>
>
> namespace LuceneTest
>
> {
>
> public partial class IndexingSetsStuck : Form
>
> {
>
> public IndexingSetsStuck()
>
> {
>
> InitializeComponent();
>
> }
>
>
>
> private void button1_Click(object sender, EventArgs e)
>
> {
>
> Thread t = new Thread(new ThreadStart(CreateIndex));
>
> t.IsBackground = true;
>
> t.Start();
>
> }
>
>
>
> private void timer1_Tick(object sender, EventArgs e)
>
> {
>
> this.Text = Count.ToString();
>
> }
>
>
>
>
>
> string INDEX = @"TestIndex";
>
> IndexWriter writer = null;
>
> object LockObject = new object();
>
> int Count = 0;
>
>
>
> void Worker()
>
> {
>
> Field f = new Field("Line", "", Field.Store.YES,
> Field.Index.TOKENIZED);
>
> Document d = new Document();
>
> d.Add(f);
>
>
>
> string line = "";
>
> for (int i = 0; i < 50000; i++)
>
> {
>
> line = i.ToString().PadLeft(12, '0') + " " +
> i.ToString("x").PadLeft(10, '0');
>
> f.SetValue(line);
>
> writer.AddDocument(d);
>
>
>
> lock (LockObject) Count++;
>
> }
>
> }
>
>
>
> void CreateIndex()
>
> {
>
> writer = new IndexWriter(INDEX, new
> Lucene.Net.Analysis.WhitespaceAnalyzer(), true);
>
>
>
> /*****!!! ?????? !!!*****/
>
> //lock (writer) //Indexing gets stuck when this line is
> uncommented
>
> {
>
> Thread[] threads = new Thread[5];
>
> for (int i = 0; i < threads.Length; i++)
>
> {
>
> threads[i] = new Thread(new ThreadStart(Worker));
>
> threads[i].IsBackground = true;
>
> threads[i].Start();
>
> }
>
> for (int i = 0; i < threads.Length; i++)
>
> {
>
> threads[i].Join();
>
> }
>
> }
>
> writer.Close(true);
>
>
>
> MessageBox.Show("Indexing Completed");
>
> }
>
> }
>
> }
>
>
>
>
>
>
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 1:26 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
>
>
> Hi Digy,
>
>
>
> here is my code for add a document to lucene
>
>
>
> internal static void AddContent(Content content)
>
> {
>
> lock (writer)
>
> {
>
> writer.Open();
>
> try
>
> {
>
> writer.AddContent(content);
>
> }
>
> finally
>
> {
>
> writer.Close();
>
> }
>
> }
>
> }
>
>
>
> No magic, just like ordinary usage.
>
>
>
> Floyd
>
>
>
>
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
>
>
> > Hi Floyd,
>
> >
>
> > Is it possible to send your indexing code?
>
> >
>
> > DIGY
>
> >
>
> > -----Original Message-----
>
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > Sent: Thursday, April 23, 2009 12:56 PM
>
> > To: lucene-net-user@incubator.apache.org
>
> > Subject: Re: Luke-0.9.2 cannot open index files
>
> >
>
> > Hello Digy,
>
> >
>
> > Actually my index files that you owned is created by Lucene.Net-2.3.2 in
>
> > trunk this morning.
>
> >
>
> > Floyd
>
> >
>
> >
>
> >
>
> > 2009/4/23 Digy <di...@gmail.com>
>
> >
>
> > > Hi Floyd,
>
> > > I am using Luke 0.9.1. Altought it has the same problem while opening
>
> > your
>
> > > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
>
> > > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
>
> > >
>
> > > DIGY
>
> > >
>
> > > -----Original Message-----
>
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > > Sent: Thursday, April 23, 2009 10:49 AM
>
> > > To: lucene-net-user@incubator.apache.org
>
> > > Subject: Re: Luke-0.9.2 cannot open index files
>
> > >
>
> > > Hi Jeroen,
>
> > > First I'm sure that close the writer after doing my job. And if not
>
> > > properly
>
> > > closed writer is the main point, why Luke-0.8.1 could open? So I guess
>
> > > there
>
> > > must something wrong or changed between Lucene.Net or Luke.
>
> > >
>
> > > I've asked the same question to the author of Luke, and he( or she)
> told
>
> > me
>
> > > that index files format mabe changed but not very sure.
>
> > >
>
> > >
>
> > >
>
> > > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
>
> > >
>
> > > > Hi,
>
> > > >
>
> > > > Just a thought:
>
> > > > Are you sure your IndexWriter was properly closed?
>
> > > > I have had the same symptoms when I forgot to call the Close()
> method.
>
> > > >
>
> > > > Jeroen
>
> > > >
>
> > > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > > > Sent: donderdag 23 april 2009 8:24
>
> > > > To: lucene-net-user@incubator.apache.org
>
> > > > Subject: Luke-0.9.2 cannot open index files
>
> > > >
>
> > > > Hi does anyone have the some problem?
>
> > > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
>
> > java
>
> > > > tool "Luke-0.9.2".
>
> > > > I tried to open the same index files with Luke-0.8.1 and it works
> fine.
>
> > > >
>
> > > > Attachment is my index files.
>
> > > > Every time I opened index files with Luke-0.9.2, Luke will show a
>
> > message
>
> > > > "Read past EOF".
>
> > > >
>
> > > > Please help on this.
>
> > > >
>
> > > >
>
> > > >
>
> > > >
>
> > >
>
> > >
>
> >
>
> >
>
>
Re: Luke-0.9.2 cannot open index files
Posted by Floyd Wu <fl...@gmail.com>.
I also found the problem is optimize index files with Luke(0.8.1) will
destroy index files. Lucene.Net could not read it after Luke optimizing it.
2009/4/23 Granroth, Neal V. <ne...@thermofisher.com>
> I have encountered sporadic problems with Luke 0.9.2 similar to what you
> describe.
>
> Two solutions that have worked for me when I encounter these problems are;
> compacting the index, and using an older version of Luke such as v0.6
>
>
> -- Neal
>
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Thursday, April 23, 2009 6:32 AM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Luke-0.9.2 cannot open index files
>
> Hi Floyd,
>
>
>
> For ex, see the sample code below: If you lock IndexWriter the indexing
> code
> gets stuck. But If you don't use the lock and let the Lucene do it
> internally, indexing goes fine. Maybe your index gets corrupted with such a
> code?
>
>
>
> DIGY.
>
>
>
>
>
> using System;
>
> using System.Collections.Generic;
>
> using System.ComponentModel;
>
> using System.Data;
>
> using System.Drawing;
>
> using System.Text;
>
> using System.Windows.Forms;
>
>
>
> using System.IO;
>
> using Lucene.Net.Analysis.Unicode;
>
> using Lucene.Net.Index;
>
> using Lucene.Net.Search;
>
> using Lucene.Net.QueryParsers;
>
> using Lucene.Net.Documents;
>
>
>
> using System.Threading;
>
>
>
>
>
>
>
>
>
> namespace LuceneTest
>
> {
>
> public partial class IndexingSetsStuck : Form
>
> {
>
> public IndexingSetsStuck()
>
> {
>
> InitializeComponent();
>
> }
>
>
>
> private void button1_Click(object sender, EventArgs e)
>
> {
>
> Thread t = new Thread(new ThreadStart(CreateIndex));
>
> t.IsBackground = true;
>
> t.Start();
>
> }
>
>
>
> private void timer1_Tick(object sender, EventArgs e)
>
> {
>
> this.Text = Count.ToString();
>
> }
>
>
>
>
>
> string INDEX = @"TestIndex";
>
> IndexWriter writer = null;
>
> object LockObject = new object();
>
> int Count = 0;
>
>
>
> void Worker()
>
> {
>
> Field f = new Field("Line", "", Field.Store.YES,
> Field.Index.TOKENIZED);
>
> Document d = new Document();
>
> d.Add(f);
>
>
>
> string line = "";
>
> for (int i = 0; i < 50000; i++)
>
> {
>
> line = i.ToString().PadLeft(12, '0') + " " +
> i.ToString("x").PadLeft(10, '0');
>
> f.SetValue(line);
>
> writer.AddDocument(d);
>
>
>
> lock (LockObject) Count++;
>
> }
>
> }
>
>
>
> void CreateIndex()
>
> {
>
> writer = new IndexWriter(INDEX, new
> Lucene.Net.Analysis.WhitespaceAnalyzer(), true);
>
>
>
> /*****!!! ?????? !!!*****/
>
> //lock (writer) //Indexing gets stuck when this line is
> uncommented
>
> {
>
> Thread[] threads = new Thread[5];
>
> for (int i = 0; i < threads.Length; i++)
>
> {
>
> threads[i] = new Thread(new ThreadStart(Worker));
>
> threads[i].IsBackground = true;
>
> threads[i].Start();
>
> }
>
> for (int i = 0; i < threads.Length; i++)
>
> {
>
> threads[i].Join();
>
> }
>
> }
>
> writer.Close(true);
>
>
>
> MessageBox.Show("Indexing Completed");
>
> }
>
> }
>
> }
>
>
>
>
>
>
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 1:26 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
>
>
> Hi Digy,
>
>
>
> here is my code for add a document to lucene
>
>
>
> internal static void AddContent(Content content)
>
> {
>
> lock (writer)
>
> {
>
> writer.Open();
>
> try
>
> {
>
> writer.AddContent(content);
>
> }
>
> finally
>
> {
>
> writer.Close();
>
> }
>
> }
>
> }
>
>
>
> No magic, just like ordinary usage.
>
>
>
> Floyd
>
>
>
>
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
>
>
> > Hi Floyd,
>
> >
>
> > Is it possible to send your indexing code?
>
> >
>
> > DIGY
>
> >
>
> > -----Original Message-----
>
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > Sent: Thursday, April 23, 2009 12:56 PM
>
> > To: lucene-net-user@incubator.apache.org
>
> > Subject: Re: Luke-0.9.2 cannot open index files
>
> >
>
> > Hello Digy,
>
> >
>
> > Actually my index files that you owned is created by Lucene.Net-2.3.2 in
>
> > trunk this morning.
>
> >
>
> > Floyd
>
> >
>
> >
>
> >
>
> > 2009/4/23 Digy <di...@gmail.com>
>
> >
>
> > > Hi Floyd,
>
> > > I am using Luke 0.9.1. Altought it has the same problem while opening
>
> > your
>
> > > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
>
> > > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
>
> > >
>
> > > DIGY
>
> > >
>
> > > -----Original Message-----
>
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > > Sent: Thursday, April 23, 2009 10:49 AM
>
> > > To: lucene-net-user@incubator.apache.org
>
> > > Subject: Re: Luke-0.9.2 cannot open index files
>
> > >
>
> > > Hi Jeroen,
>
> > > First I'm sure that close the writer after doing my job. And if not
>
> > > properly
>
> > > closed writer is the main point, why Luke-0.8.1 could open? So I guess
>
> > > there
>
> > > must something wrong or changed between Lucene.Net or Luke.
>
> > >
>
> > > I've asked the same question to the author of Luke, and he( or she)
> told
>
> > me
>
> > > that index files format mabe changed but not very sure.
>
> > >
>
> > >
>
> > >
>
> > > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
>
> > >
>
> > > > Hi,
>
> > > >
>
> > > > Just a thought:
>
> > > > Are you sure your IndexWriter was properly closed?
>
> > > > I have had the same symptoms when I forgot to call the Close()
> method.
>
> > > >
>
> > > > Jeroen
>
> > > >
>
> > > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
>
> > > > Sent: donderdag 23 april 2009 8:24
>
> > > > To: lucene-net-user@incubator.apache.org
>
> > > > Subject: Luke-0.9.2 cannot open index files
>
> > > >
>
> > > > Hi does anyone have the some problem?
>
> > > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
>
> > java
>
> > > > tool "Luke-0.9.2".
>
> > > > I tried to open the same index files with Luke-0.8.1 and it works
> fine.
>
> > > >
>
> > > > Attachment is my index files.
>
> > > > Every time I opened index files with Luke-0.9.2, Luke will show a
>
> > message
>
> > > > "Read past EOF".
>
> > > >
>
> > > > Please help on this.
>
> > > >
>
> > > >
>
> > > >
>
> > > >
>
> > >
>
> > >
>
> >
>
> >
>
>
RE: Luke-0.9.2 cannot open index files
Posted by "Granroth, Neal V." <ne...@thermofisher.com>.
I have encountered sporadic problems with Luke 0.9.2 similar to what you describe.
Two solutions that have worked for me when I encounter these problems are; compacting the index, and using an older version of Luke such as v0.6
-- Neal
-----Original Message-----
From: Digy [mailto:digydigy@gmail.com]
Sent: Thursday, April 23, 2009 6:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Luke-0.9.2 cannot open index files
Hi Floyd,
For ex, see the sample code below: If you lock IndexWriter the indexing code
gets stuck. But If you don't use the lock and let the Lucene do it
internally, indexing goes fine. Maybe your index gets corrupted with such a
code?
DIGY.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.IO;
using Lucene.Net.Analysis.Unicode;
using Lucene.Net.Index;
using Lucene.Net.Search;
using Lucene.Net.QueryParsers;
using Lucene.Net.Documents;
using System.Threading;
namespace LuceneTest
{
public partial class IndexingSetsStuck : Form
{
public IndexingSetsStuck()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
Thread t = new Thread(new ThreadStart(CreateIndex));
t.IsBackground = true;
t.Start();
}
private void timer1_Tick(object sender, EventArgs e)
{
this.Text = Count.ToString();
}
string INDEX = @"TestIndex";
IndexWriter writer = null;
object LockObject = new object();
int Count = 0;
void Worker()
{
Field f = new Field("Line", "", Field.Store.YES,
Field.Index.TOKENIZED);
Document d = new Document();
d.Add(f);
string line = "";
for (int i = 0; i < 50000; i++)
{
line = i.ToString().PadLeft(12, '0') + " " +
i.ToString("x").PadLeft(10, '0');
f.SetValue(line);
writer.AddDocument(d);
lock (LockObject) Count++;
}
}
void CreateIndex()
{
writer = new IndexWriter(INDEX, new
Lucene.Net.Analysis.WhitespaceAnalyzer(), true);
/*****!!! ?????? !!!*****/
//lock (writer) //Indexing gets stuck when this line is
uncommented
{
Thread[] threads = new Thread[5];
for (int i = 0; i < threads.Length; i++)
{
threads[i] = new Thread(new ThreadStart(Worker));
threads[i].IsBackground = true;
threads[i].Start();
}
for (int i = 0; i < threads.Length; i++)
{
threads[i].Join();
}
}
writer.Close(true);
MessageBox.Show("Indexing Completed");
}
}
}
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Thursday, April 23, 2009 1:26 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Luke-0.9.2 cannot open index files
Hi Digy,
here is my code for add a document to lucene
internal static void AddContent(Content content)
{
lock (writer)
{
writer.Open();
try
{
writer.AddContent(content);
}
finally
{
writer.Close();
}
}
}
No magic, just like ordinary usage.
Floyd
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
>
> Is it possible to send your indexing code?
>
> DIGY
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 12:56 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
> Hello Digy,
>
> Actually my index files that you owned is created by Lucene.Net-2.3.2 in
> trunk this morning.
>
> Floyd
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
> > Hi Floyd,
> > I am using Luke 0.9.1. Altought it has the same problem while opening
> your
> > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
> > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
> >
> > DIGY
> >
> > -----Original Message-----
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: Thursday, April 23, 2009 10:49 AM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: Luke-0.9.2 cannot open index files
> >
> > Hi Jeroen,
> > First I'm sure that close the writer after doing my job. And if not
> > properly
> > closed writer is the main point, why Luke-0.8.1 could open? So I guess
> > there
> > must something wrong or changed between Lucene.Net or Luke.
> >
> > I've asked the same question to the author of Luke, and he( or she) told
> me
> > that index files format mabe changed but not very sure.
> >
> >
> >
> > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
> >
> > > Hi,
> > >
> > > Just a thought:
> > > Are you sure your IndexWriter was properly closed?
> > > I have had the same symptoms when I forgot to call the Close() method.
> > >
> > > Jeroen
> > >
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > > Sent: donderdag 23 april 2009 8:24
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Luke-0.9.2 cannot open index files
> > >
> > > Hi does anyone have the some problem?
> > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
> java
> > > tool "Luke-0.9.2".
> > > I tried to open the same index files with Luke-0.8.1 and it works
fine.
> > >
> > > Attachment is my index files.
> > > Every time I opened index files with Luke-0.9.2, Luke will show a
> message
> > > "Read past EOF".
> > >
> > > Please help on this.
> > >
> > >
> > >
> > >
> >
> >
>
>
RE: Luke-0.9.2 cannot open index files
Posted by Digy <di...@gmail.com>.
Hi Floyd,
For ex, see the sample code below: If you lock IndexWriter the indexing code
gets stuck. But If you don't use the lock and let the Lucene do it
internally, indexing goes fine. Maybe your index gets corrupted with such a
code?
DIGY.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.IO;
using Lucene.Net.Analysis.Unicode;
using Lucene.Net.Index;
using Lucene.Net.Search;
using Lucene.Net.QueryParsers;
using Lucene.Net.Documents;
using System.Threading;
namespace LuceneTest
{
public partial class IndexingSetsStuck : Form
{
public IndexingSetsStuck()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
Thread t = new Thread(new ThreadStart(CreateIndex));
t.IsBackground = true;
t.Start();
}
private void timer1_Tick(object sender, EventArgs e)
{
this.Text = Count.ToString();
}
string INDEX = @"TestIndex";
IndexWriter writer = null;
object LockObject = new object();
int Count = 0;
void Worker()
{
Field f = new Field("Line", "", Field.Store.YES,
Field.Index.TOKENIZED);
Document d = new Document();
d.Add(f);
string line = "";
for (int i = 0; i < 50000; i++)
{
line = i.ToString().PadLeft(12, '0') + " " +
i.ToString("x").PadLeft(10, '0');
f.SetValue(line);
writer.AddDocument(d);
lock (LockObject) Count++;
}
}
void CreateIndex()
{
writer = new IndexWriter(INDEX, new
Lucene.Net.Analysis.WhitespaceAnalyzer(), true);
/*****!!! ?????? !!!*****/
//lock (writer) //Indexing gets stuck when this line is
uncommented
{
Thread[] threads = new Thread[5];
for (int i = 0; i < threads.Length; i++)
{
threads[i] = new Thread(new ThreadStart(Worker));
threads[i].IsBackground = true;
threads[i].Start();
}
for (int i = 0; i < threads.Length; i++)
{
threads[i].Join();
}
}
writer.Close(true);
MessageBox.Show("Indexing Completed");
}
}
}
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Thursday, April 23, 2009 1:26 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Luke-0.9.2 cannot open index files
Hi Digy,
here is my code for add a document to lucene
internal static void AddContent(Content content)
{
lock (writer)
{
writer.Open();
try
{
writer.AddContent(content);
}
finally
{
writer.Close();
}
}
}
No magic, just like ordinary usage.
Floyd
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
>
> Is it possible to send your indexing code?
>
> DIGY
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 12:56 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
> Hello Digy,
>
> Actually my index files that you owned is created by Lucene.Net-2.3.2 in
> trunk this morning.
>
> Floyd
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
> > Hi Floyd,
> > I am using Luke 0.9.1. Altought it has the same problem while opening
> your
> > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
> > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
> >
> > DIGY
> >
> > -----Original Message-----
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: Thursday, April 23, 2009 10:49 AM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: Luke-0.9.2 cannot open index files
> >
> > Hi Jeroen,
> > First I'm sure that close the writer after doing my job. And if not
> > properly
> > closed writer is the main point, why Luke-0.8.1 could open? So I guess
> > there
> > must something wrong or changed between Lucene.Net or Luke.
> >
> > I've asked the same question to the author of Luke, and he( or she) told
> me
> > that index files format mabe changed but not very sure.
> >
> >
> >
> > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
> >
> > > Hi,
> > >
> > > Just a thought:
> > > Are you sure your IndexWriter was properly closed?
> > > I have had the same symptoms when I forgot to call the Close() method.
> > >
> > > Jeroen
> > >
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > > Sent: donderdag 23 april 2009 8:24
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Luke-0.9.2 cannot open index files
> > >
> > > Hi does anyone have the some problem?
> > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
> java
> > > tool "Luke-0.9.2".
> > > I tried to open the same index files with Luke-0.8.1 and it works
fine.
> > >
> > > Attachment is my index files.
> > > Every time I opened index files with Luke-0.9.2, Luke will show a
> message
> > > "Read past EOF".
> > >
> > > Please help on this.
> > >
> > >
> > >
> > >
> >
> >
>
>
Re: Luke-0.9.2 cannot open index files
Posted by Floyd Wu <fl...@gmail.com>.
Hi Digy,
here is my code for add a document to lucene
internal static void AddContent(Content content)
{
lock (writer)
{
writer.Open();
try
{
writer.AddContent(content);
}
finally
{
writer.Close();
}
}
}
No magic, just like ordinary usage.
Floyd
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
>
> Is it possible to send your indexing code?
>
> DIGY
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 12:56 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
> Hello Digy,
>
> Actually my index files that you owned is created by Lucene.Net-2.3.2 in
> trunk this morning.
>
> Floyd
>
>
>
> 2009/4/23 Digy <di...@gmail.com>
>
> > Hi Floyd,
> > I am using Luke 0.9.1. Altought it has the same problem while opening
> your
> > index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
> > Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
> >
> > DIGY
> >
> > -----Original Message-----
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: Thursday, April 23, 2009 10:49 AM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: Luke-0.9.2 cannot open index files
> >
> > Hi Jeroen,
> > First I'm sure that close the writer after doing my job. And if not
> > properly
> > closed writer is the main point, why Luke-0.8.1 could open? So I guess
> > there
> > must something wrong or changed between Lucene.Net or Luke.
> >
> > I've asked the same question to the author of Luke, and he( or she) told
> me
> > that index files format mabe changed but not very sure.
> >
> >
> >
> > 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
> >
> > > Hi,
> > >
> > > Just a thought:
> > > Are you sure your IndexWriter was properly closed?
> > > I have had the same symptoms when I forgot to call the Close() method.
> > >
> > > Jeroen
> > >
> > > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > > Sent: donderdag 23 april 2009 8:24
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Luke-0.9.2 cannot open index files
> > >
> > > Hi does anyone have the some problem?
> > > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
> java
> > > tool "Luke-0.9.2".
> > > I tried to open the same index files with Luke-0.8.1 and it works fine.
> > >
> > > Attachment is my index files.
> > > Every time I opened index files with Luke-0.9.2, Luke will show a
> message
> > > "Read past EOF".
> > >
> > > Please help on this.
> > >
> > >
> > >
> > >
> >
> >
>
>
RE: Luke-0.9.2 cannot open index files
Posted by Digy <di...@gmail.com>.
Hi Floyd,
Is it possible to send your indexing code?
DIGY
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Thursday, April 23, 2009 12:56 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Luke-0.9.2 cannot open index files
Hello Digy,
Actually my index files that you owned is created by Lucene.Net-2.3.2 in
trunk this morning.
Floyd
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
> I am using Luke 0.9.1. Altought it has the same problem while opening your
> index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
> Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
>
> DIGY
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 10:49 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
> Hi Jeroen,
> First I'm sure that close the writer after doing my job. And if not
> properly
> closed writer is the main point, why Luke-0.8.1 could open? So I guess
> there
> must something wrong or changed between Lucene.Net or Luke.
>
> I've asked the same question to the author of Luke, and he( or she) told
me
> that index files format mabe changed but not very sure.
>
>
>
> 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
>
> > Hi,
> >
> > Just a thought:
> > Are you sure your IndexWriter was properly closed?
> > I have had the same symptoms when I forgot to call the Close() method.
> >
> > Jeroen
> >
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: donderdag 23 april 2009 8:24
> > To: lucene-net-user@incubator.apache.org
> > Subject: Luke-0.9.2 cannot open index files
> >
> > Hi does anyone have the some problem?
> > My index files is built by Lucene.Net 2.3.1 and it can't be opend by
java
> > tool "Luke-0.9.2".
> > I tried to open the same index files with Luke-0.8.1 and it works fine.
> >
> > Attachment is my index files.
> > Every time I opened index files with Luke-0.9.2, Luke will show a
message
> > "Read past EOF".
> >
> > Please help on this.
> >
> >
> >
> >
>
>
Re: Luke-0.9.2 cannot open index files
Posted by Floyd Wu <fl...@gmail.com>.
Hello Digy,
Actually my index files that you owned is created by Lucene.Net-2.3.2 in
trunk this morning.
Floyd
2009/4/23 Digy <di...@gmail.com>
> Hi Floyd,
> I am using Luke 0.9.1. Altought it has the same problem while opening your
> index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
> Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
>
> DIGY
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, April 23, 2009 10:49 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Luke-0.9.2 cannot open index files
>
> Hi Jeroen,
> First I'm sure that close the writer after doing my job. And if not
> properly
> closed writer is the main point, why Luke-0.8.1 could open? So I guess
> there
> must something wrong or changed between Lucene.Net or Luke.
>
> I've asked the same question to the author of Luke, and he( or she) told me
> that index files format mabe changed but not very sure.
>
>
>
> 2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
>
> > Hi,
> >
> > Just a thought:
> > Are you sure your IndexWriter was properly closed?
> > I have had the same symptoms when I forgot to call the Close() method.
> >
> > Jeroen
> >
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: donderdag 23 april 2009 8:24
> > To: lucene-net-user@incubator.apache.org
> > Subject: Luke-0.9.2 cannot open index files
> >
> > Hi does anyone have the some problem?
> > My index files is built by Lucene.Net 2.3.1 and it can't be opend by java
> > tool "Luke-0.9.2".
> > I tried to open the same index files with Luke-0.8.1 and it works fine.
> >
> > Attachment is my index files.
> > Every time I opened index files with Luke-0.9.2, Luke will show a message
> > "Read past EOF".
> >
> > Please help on this.
> >
> >
> >
> >
>
>
RE: Luke-0.9.2 cannot open index files
Posted by Digy <di...@gmail.com>.
Hi Floyd,
I am using Luke 0.9.1. Altought it has the same problem while opening your
index, it can open my indices created with Lucene.Net 2.3.2 smoothly.
Can you try to create the same index with Lucene.Net 2.3.2 in trunk?
DIGY
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Thursday, April 23, 2009 10:49 AM
To: lucene-net-user@incubator.apache.org
Subject: Re: Luke-0.9.2 cannot open index files
Hi Jeroen,
First I'm sure that close the writer after doing my job. And if not properly
closed writer is the main point, why Luke-0.8.1 could open? So I guess there
must something wrong or changed between Lucene.Net or Luke.
I've asked the same question to the author of Luke, and he( or she) told me
that index files format mabe changed but not very sure.
2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
> Hi,
>
> Just a thought:
> Are you sure your IndexWriter was properly closed?
> I have had the same symptoms when I forgot to call the Close() method.
>
> Jeroen
>
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: donderdag 23 april 2009 8:24
> To: lucene-net-user@incubator.apache.org
> Subject: Luke-0.9.2 cannot open index files
>
> Hi does anyone have the some problem?
> My index files is built by Lucene.Net 2.3.1 and it can't be opend by java
> tool "Luke-0.9.2".
> I tried to open the same index files with Luke-0.8.1 and it works fine.
>
> Attachment is my index files.
> Every time I opened index files with Luke-0.9.2, Luke will show a message
> "Read past EOF".
>
> Please help on this.
>
>
>
>
Re: Luke-0.9.2 cannot open index files
Posted by Floyd Wu <fl...@gmail.com>.
Hi Jeroen,
First I'm sure that close the writer after doing my job. And if not properly
closed writer is the main point, why Luke-0.8.1 could open? So I guess there
must something wrong or changed between Lucene.Net or Luke.
I've asked the same question to the author of Luke, and he( or she) told me
that index files format mabe changed but not very sure.
2009/4/23 Jeroen Lauwers <Je...@ctlo.net>
> Hi,
>
> Just a thought:
> Are you sure your IndexWriter was properly closed?
> I have had the same symptoms when I forgot to call the Close() method.
>
> Jeroen
>
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: donderdag 23 april 2009 8:24
> To: lucene-net-user@incubator.apache.org
> Subject: Luke-0.9.2 cannot open index files
>
> Hi does anyone have the some problem?
> My index files is built by Lucene.Net 2.3.1 and it can't be opend by java
> tool "Luke-0.9.2".
> I tried to open the same index files with Luke-0.8.1 and it works fine.
>
> Attachment is my index files.
> Every time I opened index files with Luke-0.9.2, Luke will show a message
> "Read past EOF".
>
> Please help on this.
>
>
>
>
RE: Luke-0.9.2 cannot open index files
Posted by Jeroen Lauwers <Je...@CTLO.NET>.
Hi,
Just a thought:
Are you sure your IndexWriter was properly closed?
I have had the same symptoms when I forgot to call the Close() method.
Jeroen
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: donderdag 23 april 2009 8:24
To: lucene-net-user@incubator.apache.org
Subject: Luke-0.9.2 cannot open index files
Hi does anyone have the some problem?
My index files is built by Lucene.Net 2.3.1 and it can't be opend by java tool "Luke-0.9.2".
I tried to open the same index files with Luke-0.8.1 and it works fine.
Attachment is my index files.
Every time I opened index files with Luke-0.9.2, Luke will show a message "Read past EOF".
Please help on this.