You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Moray McConnachie <mm...@oxford-analytica.com> on 2011/02/14 19:28:04 UTC
Re: CurruptIndexException problem
I saw a problem like this a while ago when we updated our indexing app to a later (major) version of lucene, while our searching apps stayed at an earlier version.
The problem went away when we reindexed all our data and made sure the indexing app stayed behind the searching apps in terms of the Lucene version used.
Could this apply in your case?
M.
------------------
Moray McConnachie
Director of IT,
Oxford Analytica
-----------------------------------------
Disclaimer
This message and any attachments are confidential and/or privileged. If this has been sent to you in error, please do not use, retain or disclose them, and contact the sender as soon as possible.
Oxford Analytica Ltd
Registered in England: No. 1196703
5 Alfred Street, Oxford
United Kingdom, OX1 4EH
-----------------------------------------
-----Original Message-----
From: "Digy" <di...@gmail.com>
Date: Mon, 14 Feb 2011 20:14:52
To: <lu...@lucene.apache.org>
Reply-To: lucene-net-user@lucene.apache.org
Subject: RE: CurruptIndexException problem
Hi Oded,
I just created a 3M doc. Index, made searches, inspected it with Luke 0.9.9
and everything seemed to be alright.
Index corruption is a very serious case and Lucene.Net tries to handle such
cases so that even if you reset your computer in the middle of an critical
operation you shouldn't get corrupted index( some docs not commited may get
lost).
Is it possible that your index is modified by an external process such as
virus scanners etc?
DIGY
-----Original Message-----
From: Oded Olberg [mailto:OdedO@cet.ac.il]
Sent: Sunday, February 13, 2011 11:22 AM
To: 'lucene-net-user@lucene.apache.org'
Subject: CurruptIndexException problem
I'm trying to query a large index using Lucene.NET (version 2.9.2.1).
The index contains ~1.2 Million docs and its size is 1.7GB
When trying to open an IndexSearcher, I get the following Exception:
Lucene.Net.Index.CurruptIndexException - {"Incompatible format version: 2
expected 1 or lower"}
at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)
at Lucene.Net.Index.DirectoryReader.Open(Directory directory,
IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly,
Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexReader.Open(Directory directory,
IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly,
Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexReader.Open(Directory directory, Boolean
readOnly)
at Lucene.Net.Search.IndexSearcher..ctor(Directory path, Boolean
readOnly)
at CET.KotarIndexBuilder.KotarParagraphsSearcher.Search(String
sQueryTerm) in C:\Users\odedo\documents\visual studio
2010\Projects\CET.LucenePOC\CET.KotarIndexBuilder\KotarParagraphsSearcher.cs
:line 42
at CET.KotarIndexBuilder.Program.Main(String[] args) in
C:\Users\odedo\documents\visual studio
2010\Projects\CET.LucenePOC\CET.KotarIndexBuilder\Program.cs:line 23
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[]
args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence
assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
The strange thing is, that I can open the index for writing/updating, and I
can also open the index using Luke, and query it.
I tried to optimize the index using Luke, but it did not do any good.
When I rebuild this index with a subset of the data, for example only 500K
or 750K of documents, I can open and query it successfully... but somewhere
over 1M docs, something goes wrong.
I'm running VS2010, on .NET 4.0, using 64bit machine over Win7Pro.
Thanks,
Oded
RE: CurruptIndexException problem
Posted by Oded Olberg <Od...@cet.ac.il>.
Hi all,
This was it.
After optimizing the index using Luke, I could not search or read the index.
I've rebuilt the index overnight (all the 1.5M docs) and was able to query it using Lucene.Net 2.9.2.
Thank you for your help.
Oded
-----Original Message-----
From: Hugh Spiller [mailto:Hugh.Spiller@Renishaw.com]
Sent: Tuesday, February 15, 2011 3:38 PM
To: lucene-net-user@lucene.apache.org
Subject: RE: CurruptIndexException problem
Hi Oded,
If you're editing or optimizing the index using the latest Luke (1.0.1), you're upgrading it to Luke's Lucene 3.0.1 level. Could that be the problem? Try using an older version.
Hugh
-----Original Message-----
From: Oded Olberg [mailto:OdedO@cet.ac.il]
Sent: 15 February 2011 08:15
To: lucene-net-user@lucene.apache.org
Subject: RE: CurruptIndexException problem
Both the creation and the querying attempt of the index was done using Lucene.Net 2.9.2
Thanks,
Oded
-----Original Message-----
From: Moray McConnachie [mailto:mmcconna@oxford-analytica.com]
Sent: Monday, February 14, 2011 8:28 PM
To: lucene-net-user@lucene.apache.org
Subject: Re: CurruptIndexException problem
I saw a problem like this a while ago when we updated our indexing app to a later (major) version of lucene, while our searching apps stayed at an earlier version.
The problem went away when we reindexed all our data and made sure the indexing app stayed behind the searching apps in terms of the Lucene version used.
Could this apply in your case?
M.
------------------
Moray McConnachie
Director of IT,
Oxford Analytica
-----------------------------------------
Disclaimer
This message and any attachments are confidential and/or privileged. If this has been sent to you in error, please do not use, retain or disclose them, and contact the sender as soon as possible.
Oxford Analytica Ltd
Registered in England: No. 1196703
5 Alfred Street, Oxford
United Kingdom, OX1 4EH
-----------------------------------------
-----Original Message-----
From: "Digy" <di...@gmail.com>
Date: Mon, 14 Feb 2011 20:14:52
To: <lu...@lucene.apache.org>
Reply-To: lucene-net-user@lucene.apache.org
Subject: RE: CurruptIndexException problem
Hi Oded,
I just created a 3M doc. Index, made searches, inspected it with Luke 0.9.9
and everything seemed to be alright.
Index corruption is a very serious case and Lucene.Net tries to handle such
cases so that even if you reset your computer in the middle of an critical
operation you shouldn't get corrupted index( some docs not commited may get
lost).
Is it possible that your index is modified by an external process such as
virus scanners etc?
DIGY
-----Original Message-----
From: Oded Olberg [mailto:OdedO@cet.ac.il]
Sent: Sunday, February 13, 2011 11:22 AM
To: 'lucene-net-user@lucene.apache.org'
Subject: CurruptIndexException problem
I'm trying to query a large index using Lucene.NET (version 2.9.2.1).
The index contains ~1.2 Million docs and its size is 1.7GB
When trying to open an IndexSearcher, I get the following Exception:
Lucene.Net.Index.CurruptIndexException - {"Incompatible format version: 2
expected 1 or lower"}
at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)
at Lucene.Net.Index.DirectoryReader.Open(Directory directory,
IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly,
Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexReader.Open(Directory directory,
IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly,
Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexReader.Open(Directory directory, Boolean
readOnly)
at Lucene.Net.Search.IndexSearcher..ctor(Directory path, Boolean
readOnly)
at CET.KotarIndexBuilder.KotarParagraphsSearcher.Search(String
sQueryTerm) in C:\Users\odedo\documents\visual studio
2010\Projects\CET.LucenePOC\CET.KotarIndexBuilder\KotarParagraphsSearcher.cs
:line 42
at CET.KotarIndexBuilder.Program.Main(String[] args) in
C:\Users\odedo\documents\visual studio
2010\Projects\CET.LucenePOC\CET.KotarIndexBuilder\Program.cs:line 23
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[]
args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence
assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
The strange thing is, that I can open the index for writing/updating, and I
can also open the index using Luke, and query it.
I tried to optimize the index using Luke, but it did not do any good.
When I rebuild this index with a subset of the data, for example only 500K
or 750K of documents, I can open and query it successfully... but somewhere
over 1M docs, something goes wrong.
I'm running VS2010, on .NET 4.0, using 64bit machine over Win7Pro.
Thanks,
Oded
--------------------------------------------------------------------------------------------------
This email and any attachments are confidential and are for the use of the addressee only. If you are not the addressee, you must not use or disclose the contents to any other person. Please immediately notify the sender and delete the email. Statements and opinions expressed here may not represent those of the company. Email correspondence is monitored by the company. This information may be subject to export control regulation. You are obliged to comply with such regulations.
The parent company of the Renishaw Group is Renishaw plc, registered in England no. 1106260. Registered Office: New Mills, Wotton-under-Edge, Gloucestershire, GL12 8JR, United Kingdom. Tel +44 (0) 1453 524524
--------------------------------------------------------------------------------------------------
RE: CurruptIndexException problem
Posted by Hugh Spiller <Hu...@Renishaw.com>.
Hi Oded,
If you're editing or optimizing the index using the latest Luke (1.0.1), you're upgrading it to Luke's Lucene 3.0.1 level. Could that be the problem? Try using an older version.
Hugh
-----Original Message-----
From: Oded Olberg [mailto:OdedO@cet.ac.il]
Sent: 15 February 2011 08:15
To: lucene-net-user@lucene.apache.org
Subject: RE: CurruptIndexException problem
Both the creation and the querying attempt of the index was done using Lucene.Net 2.9.2
Thanks,
Oded
-----Original Message-----
From: Moray McConnachie [mailto:mmcconna@oxford-analytica.com]
Sent: Monday, February 14, 2011 8:28 PM
To: lucene-net-user@lucene.apache.org
Subject: Re: CurruptIndexException problem
I saw a problem like this a while ago when we updated our indexing app to a later (major) version of lucene, while our searching apps stayed at an earlier version.
The problem went away when we reindexed all our data and made sure the indexing app stayed behind the searching apps in terms of the Lucene version used.
Could this apply in your case?
M.
------------------
Moray McConnachie
Director of IT,
Oxford Analytica
-----------------------------------------
Disclaimer
This message and any attachments are confidential and/or privileged. If this has been sent to you in error, please do not use, retain or disclose them, and contact the sender as soon as possible.
Oxford Analytica Ltd
Registered in England: No. 1196703
5 Alfred Street, Oxford
United Kingdom, OX1 4EH
-----------------------------------------
-----Original Message-----
From: "Digy" <di...@gmail.com>
Date: Mon, 14 Feb 2011 20:14:52
To: <lu...@lucene.apache.org>
Reply-To: lucene-net-user@lucene.apache.org
Subject: RE: CurruptIndexException problem
Hi Oded,
I just created a 3M doc. Index, made searches, inspected it with Luke 0.9.9
and everything seemed to be alright.
Index corruption is a very serious case and Lucene.Net tries to handle such
cases so that even if you reset your computer in the middle of an critical
operation you shouldn't get corrupted index( some docs not commited may get
lost).
Is it possible that your index is modified by an external process such as
virus scanners etc?
DIGY
-----Original Message-----
From: Oded Olberg [mailto:OdedO@cet.ac.il]
Sent: Sunday, February 13, 2011 11:22 AM
To: 'lucene-net-user@lucene.apache.org'
Subject: CurruptIndexException problem
I'm trying to query a large index using Lucene.NET (version 2.9.2.1).
The index contains ~1.2 Million docs and its size is 1.7GB
When trying to open an IndexSearcher, I get the following Exception:
Lucene.Net.Index.CurruptIndexException - {"Incompatible format version: 2
expected 1 or lower"}
at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)
at Lucene.Net.Index.DirectoryReader.Open(Directory directory,
IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly,
Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexReader.Open(Directory directory,
IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly,
Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexReader.Open(Directory directory, Boolean
readOnly)
at Lucene.Net.Search.IndexSearcher..ctor(Directory path, Boolean
readOnly)
at CET.KotarIndexBuilder.KotarParagraphsSearcher.Search(String
sQueryTerm) in C:\Users\odedo\documents\visual studio
2010\Projects\CET.LucenePOC\CET.KotarIndexBuilder\KotarParagraphsSearcher.cs
:line 42
at CET.KotarIndexBuilder.Program.Main(String[] args) in
C:\Users\odedo\documents\visual studio
2010\Projects\CET.LucenePOC\CET.KotarIndexBuilder\Program.cs:line 23
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[]
args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence
assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
The strange thing is, that I can open the index for writing/updating, and I
can also open the index using Luke, and query it.
I tried to optimize the index using Luke, but it did not do any good.
When I rebuild this index with a subset of the data, for example only 500K
or 750K of documents, I can open and query it successfully... but somewhere
over 1M docs, something goes wrong.
I'm running VS2010, on .NET 4.0, using 64bit machine over Win7Pro.
Thanks,
Oded
--------------------------------------------------------------------------------------------------
This email and any attachments are confidential and are for the use of the addressee only. If you are not the addressee, you must not use or disclose the contents to any other person. Please immediately notify the sender and delete the email. Statements and opinions expressed here may not represent those of the company. Email correspondence is monitored by the company. This information may be subject to export control regulation. You are obliged to comply with such regulations.
The parent company of the Renishaw Group is Renishaw plc, registered in England no. 1106260. Registered Office: New Mills, Wotton-under-Edge, Gloucestershire, GL12 8JR, United Kingdom. Tel +44 (0) 1453 524524
--------------------------------------------------------------------------------------------------
RE: CurruptIndexException problem
Posted by Oded Olberg <Od...@cet.ac.il>.
Both the creation and the querying attempt of the index was done using Lucene.Net 2.9.2
Thanks,
Oded
-----Original Message-----
From: Moray McConnachie [mailto:mmcconna@oxford-analytica.com]
Sent: Monday, February 14, 2011 8:28 PM
To: lucene-net-user@lucene.apache.org
Subject: Re: CurruptIndexException problem
I saw a problem like this a while ago when we updated our indexing app to a later (major) version of lucene, while our searching apps stayed at an earlier version.
The problem went away when we reindexed all our data and made sure the indexing app stayed behind the searching apps in terms of the Lucene version used.
Could this apply in your case?
M.
------------------
Moray McConnachie
Director of IT,
Oxford Analytica
-----------------------------------------
Disclaimer
This message and any attachments are confidential and/or privileged. If this has been sent to you in error, please do not use, retain or disclose them, and contact the sender as soon as possible.
Oxford Analytica Ltd
Registered in England: No. 1196703
5 Alfred Street, Oxford
United Kingdom, OX1 4EH
-----------------------------------------
-----Original Message-----
From: "Digy" <di...@gmail.com>
Date: Mon, 14 Feb 2011 20:14:52
To: <lu...@lucene.apache.org>
Reply-To: lucene-net-user@lucene.apache.org
Subject: RE: CurruptIndexException problem
Hi Oded,
I just created a 3M doc. Index, made searches, inspected it with Luke 0.9.9
and everything seemed to be alright.
Index corruption is a very serious case and Lucene.Net tries to handle such
cases so that even if you reset your computer in the middle of an critical
operation you shouldn't get corrupted index( some docs not commited may get
lost).
Is it possible that your index is modified by an external process such as
virus scanners etc?
DIGY
-----Original Message-----
From: Oded Olberg [mailto:OdedO@cet.ac.il]
Sent: Sunday, February 13, 2011 11:22 AM
To: 'lucene-net-user@lucene.apache.org'
Subject: CurruptIndexException problem
I'm trying to query a large index using Lucene.NET (version 2.9.2.1).
The index contains ~1.2 Million docs and its size is 1.7GB
When trying to open an IndexSearcher, I get the following Exception:
Lucene.Net.Index.CurruptIndexException - {"Incompatible format version: 2
expected 1 or lower"}
at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)
at Lucene.Net.Index.DirectoryReader.Open(Directory directory,
IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly,
Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexReader.Open(Directory directory,
IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly,
Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexReader.Open(Directory directory, Boolean
readOnly)
at Lucene.Net.Search.IndexSearcher..ctor(Directory path, Boolean
readOnly)
at CET.KotarIndexBuilder.KotarParagraphsSearcher.Search(String
sQueryTerm) in C:\Users\odedo\documents\visual studio
2010\Projects\CET.LucenePOC\CET.KotarIndexBuilder\KotarParagraphsSearcher.cs
:line 42
at CET.KotarIndexBuilder.Program.Main(String[] args) in
C:\Users\odedo\documents\visual studio
2010\Projects\CET.LucenePOC\CET.KotarIndexBuilder\Program.cs:line 23
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[]
args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence
assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
The strange thing is, that I can open the index for writing/updating, and I
can also open the index using Luke, and query it.
I tried to optimize the index using Luke, but it did not do any good.
When I rebuild this index with a subset of the data, for example only 500K
or 750K of documents, I can open and query it successfully... but somewhere
over 1M docs, something goes wrong.
I'm running VS2010, on .NET 4.0, using 64bit machine over Win7Pro.
Thanks,
Oded