You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Wai Wong (JIRA)" <ji...@apache.org> on 2009/09/01 14:22:33 UTC
[jira] Created: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Non-stored fields are not copied in writer.addDocument()?
---------------------------------------------------------
Key: LUCENE-1881
URL: https://issues.apache.org/jira/browse/LUCENE-1881
Project: Lucene - Java
Issue Type: Bug
Components: Store
Affects Versions: 2.4.1
Environment: Linux
Reporter: Wai Wong
Priority: Critical
We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
Sample code in jsp is attached:
<%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
<%@ page language="java" import="org.apache.lucene.document.*;"%>
<%@ page language="java" import="org.apache.lucene.index.*;"%>
<%@ page language="java" import="org.apache.lucene.search.*;"%>
<%@ page contentType="text/html; charset=utf8" %>
<%
// create for testing
IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
Document doc = new Document();
doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
writer.addDocument(doc);
writer.close();
// check ok
Query q = new TermQuery(new Term("A", "1234"));
Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
Hits h = s.search(q);
out.println("# of document found is " + h.length()); // it is ok
// update the document to change or remove a field
IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
doc = r.document(0);
r.deleteDocument(0);
r.close();
doc.removeField("B");
writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
writer.addDocument(doc);
writer.optimize();
writer.close();
// test again
s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
h = s.search(q);
out.println("<P># of document found is now " + h.length());
r = IndexReader.open("/opt/wwwroot/351/Index/test1");
out.println("<P> max Doc is " + r.maxDoc());
%>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750284#action_12750284 ]
Uwe Schindler commented on LUCENE-1881:
---------------------------------------
There is no practical solution for this, indexing is a one-way action and not reversible. Because of this we offer "stored" fields as a store for the orginal or additional information to the indexed documents (e.g. for storing the original strings indexed).
Lucene works with an "inverted index" ([http://en.wikipedia.org/wiki/Inverted_index]). During inversion of these non-stored fields (indexed ones), the fields are tokenized (which is a non-reversible action, because stop-words are removed, terms are normalized and so on) and these terms are stored in a global unique list off all terms. The index then only contains the references to the document ids (one-way from term -> document id). For your problem you need to get the list of terms for one document which is not easily possible (there is some possibility to iterate over all terms/docs and try to rebuild the terms for a document, but you never get back the old indexed contents and its very slow. Look into the tool "Luke" for this, which is a GUI for Lucene that has some code to do this).
You can only add your already indexed contents to another index using IndexWriter.addIndexes(). In this case they stay searchable but cannot be modified.
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Wai Wong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750296#action_12750296 ]
Wai Wong commented on LUCENE-1881:
----------------------------------
I searched a few other discussions and confirmed this behavior, and the Lazy Load feature was introduced to compensate the penalty of storing all these fields. The problem now is that the Lazy Load feature is only applicable to IndexReader, not IndexSearcher. That means I have to load all contents in searching even if I am not going to use them, just because I must keep the possibility of modifying the index db. I am indexing large number of files and this is a concern for me.
I suppose a similar method like IndexReader.document(int n, FieldSelector fieldSelector) should also be provided for IndexSearcher as IndexSearcher is much more frequently than IndexReader in most cases.
Please correct me if I am wrong.
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750210#action_12750210 ]
Hoss Man commented on LUCENE-1881:
----------------------------------
there is no bug in addDocument.
the behavior observed is a basic tenant of retrieving documents from an IndexReader...
http://lucene.apache.org/java/2_4_1/api/core/org/apache/lucene/index/IndexReader.html#document(int)
bq. Returns the stored fields of the nth Document in this index.
See also...
http://lucene.apache.org/java/2_4_1/api/core/org/apache/lucene/document/Document.html
bq. ...Note that fields which are not stored are not available in documents retrieved from the index, e.g. with ScoreDoc.doc, Searcher.doc(int) or IndexReader.document(int).
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man resolved LUCENE-1881.
------------------------------
Resolution: Fixed
Assignee: Hoss Man
Committed revision 810324.
i went ahead and fixed this using the "display the '1' bucket" approach.
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man resolved LUCENE-1881.
------------------------------
Resolution: Invalid
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Wai Wong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750315#action_12750315 ]
Wai Wong commented on LUCENE-1881:
----------------------------------
Thanks. I just checked Searcher and forgot to check IndexSearcher. I shall try this out and see if the performance is ok, especially the overhead in adding new entries and optimizing/merging the indexes.
If not, I would turn back and use my own way of buffering the non-Stored fields.
Thanks for the support.
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-1881) Non-stored fields are
not copied in writer.addDocument()?
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750211#action_12750211 ]
Hoss Man edited comment on LUCENE-1881 at 9/1/09 5:57 PM:
----------------------------------------------------------
*REDACTED* ... comment put in wrong issue
was (Author: hossman):
Committed revision 810324.
i went ahead and fixed this using the "display the '1' bucket" approach.
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750307#action_12750307 ]
Uwe Schindler commented on LUCENE-1881:
---------------------------------------
IndexSearcher has this method: [http://lucene.apache.org/java/2_4_1/api/org/apache/lucene/search/IndexSearcher.html#doc(int, org.apache.lucene.document.FieldSelector)]
Uwe
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Wai Wong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750247#action_12750247 ]
Wai Wong commented on LUCENE-1881:
----------------------------------
I re-read the comments again. Seems that this is not addressed as a bug. We know that the non-stored fields are not retrievable, but they are searchable in the original db. However, after copying to the new index db, the same field is no longer searchable. We just confirmed that a simple copy without any modification also cannot retain the information.
I can imagine maybe the information related to non-stored fields are not stored within the documents, but instead in some global area. I checked the document but could not get any hint where they are and how to copy them. Please point me to the right direction if possible.
Thanks.
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man reopened LUCENE-1881:
------------------------------
reopening to mark with correct resolution
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1881) Non-stored fields are not copied in
writer.addDocument()?
Posted by "Wai Wong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750243#action_12750243 ]
Wai Wong commented on LUCENE-1881:
----------------------------------
Thanks Hoss for this quick response. Where could I get the fix for testing? We are rushing for a roll out requiring this fix, thanks.
> Non-stored fields are not copied in writer.addDocument()?
> ---------------------------------------------------------
>
> Key: LUCENE-1881
> URL: https://issues.apache.org/jira/browse/LUCENE-1881
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4.1
> Environment: Linux
> Reporter: Wai Wong
> Assignee: Hoss Man
> Priority: Critical
>
> We would like to modified stored documents properties. The method is to use IndexReader to open all files, modified some fields, and copy the document via addDocument() of IndexWriter to another index. But all fields that are created using Field.Store.NO are no longer available for searching.
> Sample code in jsp is attached:
> <%@ page language="java" import="org.apache.lucene.analysis.standard.StandardAnalyzer;"%>
> <%@ page language="java" import="org.apache.lucene.document.*;"%>
> <%@ page language="java" import="org.apache.lucene.index.*;"%>
> <%@ page language="java" import="org.apache.lucene.search.*;"%>
> <%@ page contentType="text/html; charset=utf8" %>
> <%
> // create for testing
> IndexWriter writer = new IndexWriter("/opt/wwwroot/351/Index/test", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> Document doc = new Document();
> doc.add(new Field("A", "1234", Field.Store.NO , Field.Index.NOT_ANALYZED));
> doc.add(new Field("B", "abcd", Field.Store.NO , Field.Index.NOT_ANALYZED));
> writer.addDocument(doc);
> writer.close();
> // check ok
> Query q = new TermQuery(new Term("A", "1234"));
> Searcher s = new IndexSearcher("/opt/wwwroot/351/Index/test");
> Hits h = s.search(q);
> out.println("# of document found is " + h.length()); // it is ok
> // update the document to change or remove a field
> IndexReader r = IndexReader.open("/opt/wwwroot/351/Index/test");
> doc = r.document(0);
> r.deleteDocument(0);
> r.close();
> doc.removeField("B");
> writer = new IndexWriter("/opt/wwwroot/351/Index/test1", new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
> // test again
> s = new IndexSearcher("/opt/wwwroot/351/Index/test1");
> h = s.search(q);
> out.println("<P># of document found is now " + h.length());
> r = IndexReader.open("/opt/wwwroot/351/Index/test1");
> out.println("<P> max Doc is " + r.maxDoc());
> %>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org