You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2007/11/02 19:30:50 UTC
[jira] Created: (LUCENE-1043) Speedup merging of stored fields when
field mapping "matches"
Speedup merging of stored fields when field mapping "matches"
-------------------------------------------------------------
Key: LUCENE-1043
URL: https://issues.apache.org/jira/browse/LUCENE-1043
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.2
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Minor
Fix For: 2.3
Attachments: LUCENE-1043.patch
Robert Engels suggested the following idea, here:
http://www.gossamer-threads.com/lists/lucene/java-dev/54217
When merging in the stored fields from a segment, if the field name ->
number mapping is identical then we can simply bulk copy the entire
entry for the document rather than re-interpreting and then re-writing
the actual stored fields.
I've pulled the code from the above thread and got it working on the
current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1043) Speedup merging of stored fields
when field mapping "matches"
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539675 ]
Yonik Seeley commented on LUCENE-1043:
--------------------------------------
You're fast!
Future optimizations could include bulk copying multiple documents at once (all ranges between deleted docs). The speedup would probably be greatest for small docs, but I'm not sure if it would be worth it or not.
More controversial: maybe even expand the number of docs that can be bulk copied by not bothering removing deleted docs if it's some very small number (unless it's an optimize). This is probably not worth it.
> Speedup merging of stored fields when field mapping "matches"
> -------------------------------------------------------------
>
> Key: LUCENE-1043
> URL: https://issues.apache.org/jira/browse/LUCENE-1043
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1043.patch
>
>
> Robert Engels suggested the following idea, here:
> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
> When merging in the stored fields from a segment, if the field name ->
> number mapping is identical then we can simply bulk copy the entire
> entry for the document rather than re-interpreting and then re-writing
> the actual stored fields.
> I've pulled the code from the above thread and got it working on the
> current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1043) Speedup merging of stored fields
when field mapping "matches"
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539682 ]
Michael McCandless commented on LUCENE-1043:
--------------------------------------------
{quote}
Future optimizations could include bulk copying multiple documents at once (all ranges between deleted docs). The speedup would probably be greatest for small docs, but I'm not sure if it would be worth it or not.
{quote}
Ooh, I like that idea! I'll explore that.
{quote}
More controversial: maybe even expand the number of docs that can be bulk copied by not bothering removing deleted docs if it's some very small number (unless it's an optimize). This is probably not worth it.
{quote}
That's a neat idea too but I agree likely not worth it.
Another idea: we can *almost* just concatenate the posting lists
(frq/prx) for each term, because they are "delta coded" (we write the
delta between docIDs). The only catch is you have to "stitch up" the
boundary: you have to read the docID from the start of the next
segment, write the delta-code, then you can copy the remaining bytes.
I think this could be a big win especially when merging larger
segments.
> Speedup merging of stored fields when field mapping "matches"
> -------------------------------------------------------------
>
> Key: LUCENE-1043
> URL: https://issues.apache.org/jira/browse/LUCENE-1043
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1043.patch
>
>
> Robert Engels suggested the following idea, here:
> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
> When merging in the stored fields from a segment, if the field name ->
> number mapping is identical then we can simply bulk copy the entire
> entry for the document rather than re-interpreting and then re-writing
> the actual stored fields.
> I've pulled the code from the above thread and got it working on the
> current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1043) Speedup merging of stored fields
when field mapping "matches"
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539688 ]
Yonik Seeley commented on LUCENE-1043:
--------------------------------------
re bulk copying: Ideally, read a group of docs into a buffer big enough that it triggers the IndexInput to read directly into it, and write directly from it. The field index needs to be done int by int, but it's just adding a constant to all of them and probably isn't worth optimizing (trying to not fully encode/decode).... just loop over them ahead of time, fixing them up. The total size of the stored fields to write is simply the difference between the indicies (need to slightly special case the end of the index of course...)
{quote}Another idea: we can almost just concatenate the posting lists
(frq/prx) for each term, because they are "delta coded" (we write the
delta between docIDs)
{quote}
Nice! new JIRA issue?
> Speedup merging of stored fields when field mapping "matches"
> -------------------------------------------------------------
>
> Key: LUCENE-1043
> URL: https://issues.apache.org/jira/browse/LUCENE-1043
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1043.patch
>
>
> Robert Engels suggested the following idea, here:
> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
> When merging in the stored fields from a segment, if the field name ->
> number mapping is identical then we can simply bulk copy the entire
> entry for the document rather than re-interpreting and then re-writing
> the actual stored fields.
> I've pulled the code from the above thread and got it working on the
> current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1043) Speedup merging of stored fields
when field mapping "matches"
Posted by "robert engels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539686 ]
robert engels commented on LUCENE-1043:
---------------------------------------
You have to be careful with the bulk copy because you will need to also copy the offsets from the source, sticking them up, and then write them in bulk.
With large enough input and output buffers I am not sure the bulk copy will make that large a difference.
> Speedup merging of stored fields when field mapping "matches"
> -------------------------------------------------------------
>
> Key: LUCENE-1043
> URL: https://issues.apache.org/jira/browse/LUCENE-1043
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1043.patch
>
>
> Robert Engels suggested the following idea, here:
> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
> When merging in the stored fields from a segment, if the field name ->
> number mapping is identical then we can simply bulk copy the entire
> entry for the document rather than re-interpreting and then re-writing
> the actual stored fields.
> I've pulled the code from the above thread and got it working on the
> current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1043) Speedup merging of stored fields
when field mapping "matches"
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1043.
----------------------------------------
Resolution: Fixed
I just committed this. Thanks Robert!
> Speedup merging of stored fields when field mapping "matches"
> -------------------------------------------------------------
>
> Key: LUCENE-1043
> URL: https://issues.apache.org/jira/browse/LUCENE-1043
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1043.patch, LUCENE-1043.take2.patch
>
>
> Robert Engels suggested the following idea, here:
> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
> When merging in the stored fields from a segment, if the field name ->
> number mapping is identical then we can simply bulk copy the entire
> entry for the document rather than re-interpreting and then re-writing
> the actual stored fields.
> I've pulled the code from the above thread and got it working on the
> current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Posted by robert engels <re...@ix.netcom.com>.
Here is the fixed code. Need to update the correct number of
documents... duh.
SegmentMerger.java
try {
for (int i = 0; i < readers.size(); i++) {
IndexReader reader = (IndexReader) readers.elementAt(i);
SegmentReader sreader = sreaders[i];
int maxDoc = reader.maxDoc();
for (int j = 0; j < maxDoc;)
if (!reader.isDeleted(j)) { // skip deleted docs
if (sreader!=null) {
int start=j;
int ndocs=1;
for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j+
+,ndocs++);
ndocs = sreader.documents(buffer,lengths,start,j);
fieldsWriter.addDocuments(buffer,lengths,ndocs);
docCount+=ndocs;
} else {
fieldsWriter.addDocument(reader.document(j));
j++;
docCount++;
}
}
}
} finally {
fieldsWriter.close();
}
On Nov 2, 2007, at 4:56 PM, robert engels wrote:
> Oops.. something else is wrong. I get a "docs out of order"
> exception when building a large index.
>
> Tracking it down...
>
> On Nov 2, 2007, at 4:54 PM, robert engels wrote:
>
>> Should be "... LAST document in a segment." I need to slow down.
>>
>> On Nov 2, 2007, at 4:49 PM, robert engels wrote:
>>
>>> Here is the fixed code - need special handling when determining
>>> the length of the document in a segment.
>>>
>>> FieldsReader.java
>>>
>>> final int docs(byte[] buffer,int[] lengths,int start,int max)
>>> throws IOException {
>>> final IndexInput indexStream = (IndexInput) indexLocal.get();
>>> indexStream.seek(start * 8L);
>>> long startoffset = indexStream.readLong();
>>> long lastoffset = startoffset;
>>> int count=0;
>>> while(count<max) {
>>> long offset = (start + count + 1 < size) ? indexStream.readLong
>>> () : fieldsStream.length();
>>> if(offset-startoffset>buffer.length)
>>> break;
>>> lengths[count]=(int) (offset-lastoffset);
>>> lastoffset = offset;
>>> count++;
>>> }
>>> if(count==0)
>>> throw new IllegalStateException("buffer too small for a single
>>> document");
>>>
>>> final IndexInput fieldsStream = (IndexInput) fieldsLocal.get
>>> ();
>>> fieldsStream.seek(startoffset);
>>> fieldsStream.readBytes(buffer,0,(int) (lastoffset-
>>> startoffset));
>>>
>>> return count;
>>> }
>>>
>>>
>>> On Nov 2, 2007, at 4:33 PM, robert engels wrote:
>>>
>>>> Oops. Be careful with that code, merging works but indexing
>>>> fails... I have a bug lurking...
>>>>
>>>> On Nov 2, 2007, at 4:28 PM, robert engels wrote:
>>>>
>>>>> As expected it did not make much difference. Here is the code:
>>>>>
>>>>> FieldsReader.java
>>>>> final int docs(byte[] buffer,int[] lengths,int start,int max)
>>>>> throws IOException {
>>>>> final IndexInput indexStream = (IndexInput) indexLocal.get
>>>>> ();
>>>>> indexStream.seek(start * 8L);
>>>>> long startoffset = indexStream.readLong();
>>>>> long lastoffset = startoffset;
>>>>> int count=0;
>>>>> while(count<max) {
>>>>> long offset = indexStream.readLong();
>>>>> if(offset-startoffset>buffer.length)
>>>>> break;
>>>>> lengths[count]=(int) (offset-lastoffset);
>>>>> lastoffset = offset;
>>>>> count++;
>>>>> }
>>>>> if(count==0)
>>>>> throw new IllegalStateException("buffer too small for a
>>>>> single document");
>>>>>
>>>>> final IndexInput fieldsStream = (IndexInput)
>>>>> fieldsLocal.get();
>>>>> fieldsStream.seek(startoffset);
>>>>> fieldsStream.readBytes(buffer,0,(int) (lastoffset-
>>>>> startoffset));
>>>>>
>>>>> return count;
>>>>> }
>>>>>
>>>>> FieldsWriter.java
>>>>> final void addDocuments(byte[] buffer, int[] lengths, int
>>>>> ndocs) throws IOException {
>>>>> long position = fieldsStream.getFilePointer();
>>>>> long start = position;
>>>>> for(int i=0;i<ndocs;i++) {
>>>>> indexStream.writeLong(position);
>>>>> position+=lengths[i];
>>>>> }
>>>>> fieldsStream.writeBytes(buffer,(int) (position-start));
>>>>> }
>>>>>
>>>>> SegmentReader.java
>>>>> public int documents(byte[] buffer,int[] lengths,int
>>>>> start,int max) throws IOException {
>>>>> return fieldsReader.docs(buffer,lengths,start,max);
>>>>> }
>>>>>
>>>>> SegmentMerger.java
>>>>> private final int mergeFields() throws IOException {
>>>>> fieldInfos = new FieldInfos(); // merge field names
>>>>> int docCount = 0;
>>>>> for (int i = 0; i < readers.size(); i++) {
>>>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>>>> if (reader instanceof SegmentReader) {
>>>>> SegmentReader sreader = (SegmentReader) reader;
>>>>> for (int j = 0; j < sreader.getFieldInfos().size(); j++) {
>>>>> FieldInfo fi = sreader.getFieldInfos().fieldInfo(j);
>>>>> fieldInfos.add(fi.name, fi.isIndexed, fi.storeTermVector,
>>>>> fi.storePositionWithTermVector, fi.storeOffsetWithTermVector, !
>>>>> reader.hasNorms(fi.name));
>>>>> }
>>>>> } else {
>>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION_OFFSET),
>>>>> true, true, true);
>>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION), true, true,
>>>>> false);
>>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_OFFSET), true, false,
>>>>> true);
>>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>>> (IndexReader.FieldOption.TERMVECTOR), true, false, false);
>>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>>> (IndexReader.FieldOption.INDEXED), false, false, false);
>>>>> fieldInfos.add(reader.getFieldNames
>>>>> (IndexReader.FieldOption.UNINDEXED), false);
>>>>> }
>>>>> }
>>>>> fieldInfos.write(directory, segment + ".fnm");
>>>>>
>>>>> SegmentReader[] sreaders = new SegmentReader[readers.size()];
>>>>> for (int i = 0; i < readers.size(); i++) {
>>>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>>>> boolean same = reader.getFieldNames().size() ==
>>>>> fieldInfos.size() && reader instanceof SegmentReader;
>>>>> if(same) {
>>>>> SegmentReader sreader = (SegmentReader) reader;
>>>>> for (int j = 0; same && j < fieldInfos.size(); j++) {
>>>>> same = fieldInfos.fieldName(j).equals
>>>>> (sreader.getFieldInfos().fieldName(j));
>>>>> }
>>>>> if(same)
>>>>> sreaders[i] = sreader;
>>>>> }
>>>>> }
>>>>>
>>>>> byte[] buffer = new byte[512*1024];
>>>>> int[] lengths = new int[10000];
>>>>>
>>>>> // merge field values
>>>>> FieldsWriter fieldsWriter = new FieldsWriter(directory,
>>>>> segment, fieldInfos);
>>>>>
>>>>> try {
>>>>> for (int i = 0; i < readers.size(); i++) {
>>>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>>>> SegmentReader sreader = sreaders[i];
>>>>> int maxDoc = reader.maxDoc();
>>>>> for (int j = 0; j < maxDoc;)
>>>>> if (!reader.isDeleted(j)) { // skip deleted docs
>>>>> if (sreader!=null) {
>>>>> int start=j;
>>>>> int ndocs=1;
>>>>> for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j
>>>>> ++,ndocs++);
>>>>> ndocs = sreader.documents(buffer,lengths,start,j);
>>>>> fieldsWriter.addDocuments(buffer,lengths,ndocs);
>>>>> } else {
>>>>> fieldsWriter.addDocument(reader.document(j));
>>>>> j++;
>>>>> }
>>>>> docCount++;
>>>>> }
>>>>> }
>>>>> } finally {
>>>>> fieldsWriter.close();
>>>>> }
>>>>> return docCount;
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Nov 2, 2007, at 3:49 PM, robert engels wrote:
>>>>>
>>>>>> I am working on a bulk document copy right now - I will let
>>>>>> you know if it improves things much.
>>>>>>
>>>>>> I doubt it, because I already configure the streams to use
>>>>>> fairly large input and output buffers during a merge - but the
>>>>>> memory index merge may see additional benefits due to lower
>>>>>> CPU calls.
>>>>>>
>>>>>>
>>>>>> On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
>>>>>>
>>>>>>>
>>>>>>> [ https://issues.apache.org/jira/browse/LUCENE-1043?
>>>>>>> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>>>>>>> tabpanel#action_12539696 ]
>>>>>>>
>>>>>>> robert engels commented on LUCENE-1043:
>>>>>>> ---------------------------------------
>>>>>>>
>>>>>>> When bulk copying the documents, I think you need to:
>>>>>>>
>>>>>>> read array of long from index (8 * (ndocs+1)) in long[ndocs
>>>>>>> +1] offsets;
>>>>>>> calculate length = offset[ndocs]-offset[0];
>>>>>>> read bytes of length from document file
>>>>>>> startoffset = current output document stream position
>>>>>>> write bytes to output document
>>>>>>> modify offset[] adding startoffset-offset[0] to each entry
>>>>>>> write offset[] in bulk to index output
>>>>>>>
>>>>>>>> Speedup merging of stored fields when field mapping "matches"
>>>>>>>> -------------------------------------------------------------
>>>>>>>>
>>>>>>>> Key: LUCENE-1043
>>>>>>>> URL: https://issues.apache.org/jira/browse/
>>>>>>>> LUCENE-1043
>>>>>>>> Project: Lucene - Java
>>>>>>>> Issue Type: Improvement
>>>>>>>> Components: Index
>>>>>>>> Affects Versions: 2.2
>>>>>>>> Reporter: Michael McCandless
>>>>>>>> Assignee: Michael McCandless
>>>>>>>> Priority: Minor
>>>>>>>> Fix For: 2.3
>>>>>>>>
>>>>>>>> Attachments: LUCENE-1043.patch
>>>>>>>>
>>>>>>>>
>>>>>>>> Robert Engels suggested the following idea, here:
>>>>>>>> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>>>>>>>> When merging in the stored fields from a segment, if the
>>>>>>>> field name ->
>>>>>>>> number mapping is identical then we can simply bulk copy the
>>>>>>>> entire
>>>>>>>> entry for the document rather than re-interpreting and then
>>>>>>>> re-writing
>>>>>>>> the actual stored fields.
>>>>>>>> I've pulled the code from the above thread and got it
>>>>>>>> working on the
>>>>>>>> current trunk.
>>>>>>>
>>>>>>> --
>>>>>>> This message is automatically generated by JIRA.
>>>>>>> -
>>>>>>> You can reply to this email to add a comment to the issue
>>>>>>> online.
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> -----
>>>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------------------
>>>>>> ----
>>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Posted by robert engels <re...@ix.netcom.com>.
Oops.. something else is wrong. I get a "docs out of order" exception
when building a large index.
Tracking it down...
On Nov 2, 2007, at 4:54 PM, robert engels wrote:
> Should be "... LAST document in a segment." I need to slow down.
>
> On Nov 2, 2007, at 4:49 PM, robert engels wrote:
>
>> Here is the fixed code - need special handling when determining
>> the length of the document in a segment.
>>
>> FieldsReader.java
>>
>> final int docs(byte[] buffer,int[] lengths,int start,int max)
>> throws IOException {
>> final IndexInput indexStream = (IndexInput) indexLocal.get();
>> indexStream.seek(start * 8L);
>> long startoffset = indexStream.readLong();
>> long lastoffset = startoffset;
>> int count=0;
>> while(count<max) {
>> long offset = (start + count + 1 < size) ? indexStream.readLong
>> () : fieldsStream.length();
>> if(offset-startoffset>buffer.length)
>> break;
>> lengths[count]=(int) (offset-lastoffset);
>> lastoffset = offset;
>> count++;
>> }
>> if(count==0)
>> throw new IllegalStateException("buffer too small for a single
>> document");
>>
>> final IndexInput fieldsStream = (IndexInput) fieldsLocal.get();
>> fieldsStream.seek(startoffset);
>> fieldsStream.readBytes(buffer,0,(int) (lastoffset-
>> startoffset));
>>
>> return count;
>> }
>>
>>
>> On Nov 2, 2007, at 4:33 PM, robert engels wrote:
>>
>>> Oops. Be careful with that code, merging works but indexing
>>> fails... I have a bug lurking...
>>>
>>> On Nov 2, 2007, at 4:28 PM, robert engels wrote:
>>>
>>>> As expected it did not make much difference. Here is the code:
>>>>
>>>> FieldsReader.java
>>>> final int docs(byte[] buffer,int[] lengths,int start,int max)
>>>> throws IOException {
>>>> final IndexInput indexStream = (IndexInput) indexLocal.get();
>>>> indexStream.seek(start * 8L);
>>>> long startoffset = indexStream.readLong();
>>>> long lastoffset = startoffset;
>>>> int count=0;
>>>> while(count<max) {
>>>> long offset = indexStream.readLong();
>>>> if(offset-startoffset>buffer.length)
>>>> break;
>>>> lengths[count]=(int) (offset-lastoffset);
>>>> lastoffset = offset;
>>>> count++;
>>>> }
>>>> if(count==0)
>>>> throw new IllegalStateException("buffer too small for a
>>>> single document");
>>>>
>>>> final IndexInput fieldsStream = (IndexInput)
>>>> fieldsLocal.get();
>>>> fieldsStream.seek(startoffset);
>>>> fieldsStream.readBytes(buffer,0,(int) (lastoffset-
>>>> startoffset));
>>>>
>>>> return count;
>>>> }
>>>>
>>>> FieldsWriter.java
>>>> final void addDocuments(byte[] buffer, int[] lengths, int
>>>> ndocs) throws IOException {
>>>> long position = fieldsStream.getFilePointer();
>>>> long start = position;
>>>> for(int i=0;i<ndocs;i++) {
>>>> indexStream.writeLong(position);
>>>> position+=lengths[i];
>>>> }
>>>> fieldsStream.writeBytes(buffer,(int) (position-start));
>>>> }
>>>>
>>>> SegmentReader.java
>>>> public int documents(byte[] buffer,int[] lengths,int start,int
>>>> max) throws IOException {
>>>> return fieldsReader.docs(buffer,lengths,start,max);
>>>> }
>>>>
>>>> SegmentMerger.java
>>>> private final int mergeFields() throws IOException {
>>>> fieldInfos = new FieldInfos(); // merge field names
>>>> int docCount = 0;
>>>> for (int i = 0; i < readers.size(); i++) {
>>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>>> if (reader instanceof SegmentReader) {
>>>> SegmentReader sreader = (SegmentReader) reader;
>>>> for (int j = 0; j < sreader.getFieldInfos().size(); j++) {
>>>> FieldInfo fi = sreader.getFieldInfos().fieldInfo(j);
>>>> fieldInfos.add(fi.name, fi.isIndexed, fi.storeTermVector,
>>>> fi.storePositionWithTermVector, fi.storeOffsetWithTermVector, !
>>>> reader.hasNorms(fi.name));
>>>> }
>>>> } else {
>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION_OFFSET), true,
>>>> true, true);
>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION), true, true,
>>>> false);
>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_OFFSET), true, false,
>>>> true);
>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>> (IndexReader.FieldOption.TERMVECTOR), true, false, false);
>>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>>> (IndexReader.FieldOption.INDEXED), false, false, false);
>>>> fieldInfos.add(reader.getFieldNames
>>>> (IndexReader.FieldOption.UNINDEXED), false);
>>>> }
>>>> }
>>>> fieldInfos.write(directory, segment + ".fnm");
>>>>
>>>> SegmentReader[] sreaders = new SegmentReader[readers.size()];
>>>> for (int i = 0; i < readers.size(); i++) {
>>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>>> boolean same = reader.getFieldNames().size() ==
>>>> fieldInfos.size() && reader instanceof SegmentReader;
>>>> if(same) {
>>>> SegmentReader sreader = (SegmentReader) reader;
>>>> for (int j = 0; same && j < fieldInfos.size(); j++) {
>>>> same = fieldInfos.fieldName(j).equals(sreader.getFieldInfos
>>>> ().fieldName(j));
>>>> }
>>>> if(same)
>>>> sreaders[i] = sreader;
>>>> }
>>>> }
>>>>
>>>> byte[] buffer = new byte[512*1024];
>>>> int[] lengths = new int[10000];
>>>>
>>>> // merge field values
>>>> FieldsWriter fieldsWriter = new FieldsWriter(directory,
>>>> segment, fieldInfos);
>>>>
>>>> try {
>>>> for (int i = 0; i < readers.size(); i++) {
>>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>>> SegmentReader sreader = sreaders[i];
>>>> int maxDoc = reader.maxDoc();
>>>> for (int j = 0; j < maxDoc;)
>>>> if (!reader.isDeleted(j)) { // skip deleted docs
>>>> if (sreader!=null) {
>>>> int start=j;
>>>> int ndocs=1;
>>>> for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j+
>>>> +,ndocs++);
>>>> ndocs = sreader.documents(buffer,lengths,start,j);
>>>> fieldsWriter.addDocuments(buffer,lengths,ndocs);
>>>> } else {
>>>> fieldsWriter.addDocument(reader.document(j));
>>>> j++;
>>>> }
>>>> docCount++;
>>>> }
>>>> }
>>>> } finally {
>>>> fieldsWriter.close();
>>>> }
>>>> return docCount;
>>>> }
>>>>
>>>>
>>>>
>>>>
>>>> On Nov 2, 2007, at 3:49 PM, robert engels wrote:
>>>>
>>>>> I am working on a bulk document copy right now - I will let you
>>>>> know if it improves things much.
>>>>>
>>>>> I doubt it, because I already configure the streams to use
>>>>> fairly large input and output buffers during a merge - but the
>>>>> memory index merge may see additional benefits due to lower CPU
>>>>> calls.
>>>>>
>>>>>
>>>>> On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
>>>>>
>>>>>>
>>>>>> [ https://issues.apache.org/jira/browse/LUCENE-1043?
>>>>>> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>>>>>> tabpanel#action_12539696 ]
>>>>>>
>>>>>> robert engels commented on LUCENE-1043:
>>>>>> ---------------------------------------
>>>>>>
>>>>>> When bulk copying the documents, I think you need to:
>>>>>>
>>>>>> read array of long from index (8 * (ndocs+1)) in long[ndocs+1]
>>>>>> offsets;
>>>>>> calculate length = offset[ndocs]-offset[0];
>>>>>> read bytes of length from document file
>>>>>> startoffset = current output document stream position
>>>>>> write bytes to output document
>>>>>> modify offset[] adding startoffset-offset[0] to each entry
>>>>>> write offset[] in bulk to index output
>>>>>>
>>>>>>> Speedup merging of stored fields when field mapping "matches"
>>>>>>> -------------------------------------------------------------
>>>>>>>
>>>>>>> Key: LUCENE-1043
>>>>>>> URL: https://issues.apache.org/jira/browse/
>>>>>>> LUCENE-1043
>>>>>>> Project: Lucene - Java
>>>>>>> Issue Type: Improvement
>>>>>>> Components: Index
>>>>>>> Affects Versions: 2.2
>>>>>>> Reporter: Michael McCandless
>>>>>>> Assignee: Michael McCandless
>>>>>>> Priority: Minor
>>>>>>> Fix For: 2.3
>>>>>>>
>>>>>>> Attachments: LUCENE-1043.patch
>>>>>>>
>>>>>>>
>>>>>>> Robert Engels suggested the following idea, here:
>>>>>>> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>>>>>>> When merging in the stored fields from a segment, if the
>>>>>>> field name ->
>>>>>>> number mapping is identical then we can simply bulk copy the
>>>>>>> entire
>>>>>>> entry for the document rather than re-interpreting and then
>>>>>>> re-writing
>>>>>>> the actual stored fields.
>>>>>>> I've pulled the code from the above thread and got it working
>>>>>>> on the
>>>>>>> current trunk.
>>>>>>
>>>>>> --
>>>>>> This message is automatically generated by JIRA.
>>>>>> -
>>>>>> You can reply to this email to add a comment to the issue online.
>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------------------
>>>>>> ----
>>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> ---
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>
>>>>
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Posted by robert engels <re...@ix.netcom.com>.
Should be "... LAST document in a segment." I need to slow down.
On Nov 2, 2007, at 4:49 PM, robert engels wrote:
> Here is the fixed code - need special handling when determining the
> length of the document in a segment.
>
> FieldsReader.java
>
> final int docs(byte[] buffer,int[] lengths,int start,int max)
> throws IOException {
> final IndexInput indexStream = (IndexInput) indexLocal.get();
> indexStream.seek(start * 8L);
> long startoffset = indexStream.readLong();
> long lastoffset = startoffset;
> int count=0;
> while(count<max) {
> long offset = (start + count + 1 < size) ? indexStream.readLong
> () : fieldsStream.length();
> if(offset-startoffset>buffer.length)
> break;
> lengths[count]=(int) (offset-lastoffset);
> lastoffset = offset;
> count++;
> }
> if(count==0)
> throw new IllegalStateException("buffer too small for a single
> document");
>
> final IndexInput fieldsStream = (IndexInput) fieldsLocal.get();
> fieldsStream.seek(startoffset);
> fieldsStream.readBytes(buffer,0,(int) (lastoffset-startoffset));
>
> return count;
> }
>
>
> On Nov 2, 2007, at 4:33 PM, robert engels wrote:
>
>> Oops. Be careful with that code, merging works but indexing
>> fails... I have a bug lurking...
>>
>> On Nov 2, 2007, at 4:28 PM, robert engels wrote:
>>
>>> As expected it did not make much difference. Here is the code:
>>>
>>> FieldsReader.java
>>> final int docs(byte[] buffer,int[] lengths,int start,int max)
>>> throws IOException {
>>> final IndexInput indexStream = (IndexInput) indexLocal.get();
>>> indexStream.seek(start * 8L);
>>> long startoffset = indexStream.readLong();
>>> long lastoffset = startoffset;
>>> int count=0;
>>> while(count<max) {
>>> long offset = indexStream.readLong();
>>> if(offset-startoffset>buffer.length)
>>> break;
>>> lengths[count]=(int) (offset-lastoffset);
>>> lastoffset = offset;
>>> count++;
>>> }
>>> if(count==0)
>>> throw new IllegalStateException("buffer too small for a single
>>> document");
>>>
>>> final IndexInput fieldsStream = (IndexInput) fieldsLocal.get
>>> ();
>>> fieldsStream.seek(startoffset);
>>> fieldsStream.readBytes(buffer,0,(int) (lastoffset-
>>> startoffset));
>>>
>>> return count;
>>> }
>>>
>>> FieldsWriter.java
>>> final void addDocuments(byte[] buffer, int[] lengths, int
>>> ndocs) throws IOException {
>>> long position = fieldsStream.getFilePointer();
>>> long start = position;
>>> for(int i=0;i<ndocs;i++) {
>>> indexStream.writeLong(position);
>>> position+=lengths[i];
>>> }
>>> fieldsStream.writeBytes(buffer,(int) (position-start));
>>> }
>>>
>>> SegmentReader.java
>>> public int documents(byte[] buffer,int[] lengths,int start,int
>>> max) throws IOException {
>>> return fieldsReader.docs(buffer,lengths,start,max);
>>> }
>>>
>>> SegmentMerger.java
>>> private final int mergeFields() throws IOException {
>>> fieldInfos = new FieldInfos(); // merge field names
>>> int docCount = 0;
>>> for (int i = 0; i < readers.size(); i++) {
>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>> if (reader instanceof SegmentReader) {
>>> SegmentReader sreader = (SegmentReader) reader;
>>> for (int j = 0; j < sreader.getFieldInfos().size(); j++) {
>>> FieldInfo fi = sreader.getFieldInfos().fieldInfo(j);
>>> fieldInfos.add(fi.name, fi.isIndexed, fi.storeTermVector,
>>> fi.storePositionWithTermVector, fi.storeOffsetWithTermVector, !
>>> reader.hasNorms(fi.name));
>>> }
>>> } else {
>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION_OFFSET), true,
>>> true, true);
>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION), true, true,
>>> false);
>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>> (IndexReader.FieldOption.TERMVECTOR_WITH_OFFSET), true, false,
>>> true);
>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>> (IndexReader.FieldOption.TERMVECTOR), true, false, false);
>>> addIndexed(reader, fieldInfos, reader.getFieldNames
>>> (IndexReader.FieldOption.INDEXED), false, false, false);
>>> fieldInfos.add(reader.getFieldNames
>>> (IndexReader.FieldOption.UNINDEXED), false);
>>> }
>>> }
>>> fieldInfos.write(directory, segment + ".fnm");
>>>
>>> SegmentReader[] sreaders = new SegmentReader[readers.size()];
>>> for (int i = 0; i < readers.size(); i++) {
>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>> boolean same = reader.getFieldNames().size() ==
>>> fieldInfos.size() && reader instanceof SegmentReader;
>>> if(same) {
>>> SegmentReader sreader = (SegmentReader) reader;
>>> for (int j = 0; same && j < fieldInfos.size(); j++) {
>>> same = fieldInfos.fieldName(j).equals(sreader.getFieldInfos
>>> ().fieldName(j));
>>> }
>>> if(same)
>>> sreaders[i] = sreader;
>>> }
>>> }
>>>
>>> byte[] buffer = new byte[512*1024];
>>> int[] lengths = new int[10000];
>>>
>>> // merge field values
>>> FieldsWriter fieldsWriter = new FieldsWriter(directory, segment,
>>> fieldInfos);
>>>
>>> try {
>>> for (int i = 0; i < readers.size(); i++) {
>>> IndexReader reader = (IndexReader) readers.elementAt(i);
>>> SegmentReader sreader = sreaders[i];
>>> int maxDoc = reader.maxDoc();
>>> for (int j = 0; j < maxDoc;)
>>> if (!reader.isDeleted(j)) { // skip deleted docs
>>> if (sreader!=null) {
>>> int start=j;
>>> int ndocs=1;
>>> for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j+
>>> +,ndocs++);
>>> ndocs = sreader.documents(buffer,lengths,start,j);
>>> fieldsWriter.addDocuments(buffer,lengths,ndocs);
>>> } else {
>>> fieldsWriter.addDocument(reader.document(j));
>>> j++;
>>> }
>>> docCount++;
>>> }
>>> }
>>> } finally {
>>> fieldsWriter.close();
>>> }
>>> return docCount;
>>> }
>>>
>>>
>>>
>>>
>>> On Nov 2, 2007, at 3:49 PM, robert engels wrote:
>>>
>>>> I am working on a bulk document copy right now - I will let you
>>>> know if it improves things much.
>>>>
>>>> I doubt it, because I already configure the streams to use
>>>> fairly large input and output buffers during a merge - but the
>>>> memory index merge may see additional benefits due to lower CPU
>>>> calls.
>>>>
>>>>
>>>> On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
>>>>
>>>>>
>>>>> [ https://issues.apache.org/jira/browse/LUCENE-1043?
>>>>> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>>>>> tabpanel#action_12539696 ]
>>>>>
>>>>> robert engels commented on LUCENE-1043:
>>>>> ---------------------------------------
>>>>>
>>>>> When bulk copying the documents, I think you need to:
>>>>>
>>>>> read array of long from index (8 * (ndocs+1)) in long[ndocs+1]
>>>>> offsets;
>>>>> calculate length = offset[ndocs]-offset[0];
>>>>> read bytes of length from document file
>>>>> startoffset = current output document stream position
>>>>> write bytes to output document
>>>>> modify offset[] adding startoffset-offset[0] to each entry
>>>>> write offset[] in bulk to index output
>>>>>
>>>>>> Speedup merging of stored fields when field mapping "matches"
>>>>>> -------------------------------------------------------------
>>>>>>
>>>>>> Key: LUCENE-1043
>>>>>> URL: https://issues.apache.org/jira/browse/
>>>>>> LUCENE-1043
>>>>>> Project: Lucene - Java
>>>>>> Issue Type: Improvement
>>>>>> Components: Index
>>>>>> Affects Versions: 2.2
>>>>>> Reporter: Michael McCandless
>>>>>> Assignee: Michael McCandless
>>>>>> Priority: Minor
>>>>>> Fix For: 2.3
>>>>>>
>>>>>> Attachments: LUCENE-1043.patch
>>>>>>
>>>>>>
>>>>>> Robert Engels suggested the following idea, here:
>>>>>> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>>>>>> When merging in the stored fields from a segment, if the field
>>>>>> name ->
>>>>>> number mapping is identical then we can simply bulk copy the
>>>>>> entire
>>>>>> entry for the document rather than re-interpreting and then re-
>>>>>> writing
>>>>>> the actual stored fields.
>>>>>> I've pulled the code from the above thread and got it working
>>>>>> on the
>>>>>> current trunk.
>>>>>
>>>>> --
>>>>> This message is automatically generated by JIRA.
>>>>> -
>>>>> You can reply to this email to add a comment to the issue online.
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> ---
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Posted by robert engels <re...@ix.netcom.com>.
Here is the fixed code - need special handling when determining the
length of the document in a segment.
FieldsReader.java
final int docs(byte[] buffer,int[] lengths,int start,int max)
throws IOException {
final IndexInput indexStream = (IndexInput) indexLocal.get();
indexStream.seek(start * 8L);
long startoffset = indexStream.readLong();
long lastoffset = startoffset;
int count=0;
while(count<max) {
long offset = (start + count + 1 < size) ? indexStream.readLong
() : fieldsStream.length();
if(offset-startoffset>buffer.length)
break;
lengths[count]=(int) (offset-lastoffset);
lastoffset = offset;
count++;
}
if(count==0)
throw new IllegalStateException("buffer too small for a single
document");
final IndexInput fieldsStream = (IndexInput) fieldsLocal.get();
fieldsStream.seek(startoffset);
fieldsStream.readBytes(buffer,0,(int) (lastoffset-startoffset));
return count;
}
On Nov 2, 2007, at 4:33 PM, robert engels wrote:
> Oops. Be careful with that code, merging works but indexing
> fails... I have a bug lurking...
>
> On Nov 2, 2007, at 4:28 PM, robert engels wrote:
>
>> As expected it did not make much difference. Here is the code:
>>
>> FieldsReader.java
>> final int docs(byte[] buffer,int[] lengths,int start,int max)
>> throws IOException {
>> final IndexInput indexStream = (IndexInput) indexLocal.get();
>> indexStream.seek(start * 8L);
>> long startoffset = indexStream.readLong();
>> long lastoffset = startoffset;
>> int count=0;
>> while(count<max) {
>> long offset = indexStream.readLong();
>> if(offset-startoffset>buffer.length)
>> break;
>> lengths[count]=(int) (offset-lastoffset);
>> lastoffset = offset;
>> count++;
>> }
>> if(count==0)
>> throw new IllegalStateException("buffer too small for a single
>> document");
>>
>> final IndexInput fieldsStream = (IndexInput) fieldsLocal.get();
>> fieldsStream.seek(startoffset);
>> fieldsStream.readBytes(buffer,0,(int) (lastoffset-
>> startoffset));
>>
>> return count;
>> }
>>
>> FieldsWriter.java
>> final void addDocuments(byte[] buffer, int[] lengths, int
>> ndocs) throws IOException {
>> long position = fieldsStream.getFilePointer();
>> long start = position;
>> for(int i=0;i<ndocs;i++) {
>> indexStream.writeLong(position);
>> position+=lengths[i];
>> }
>> fieldsStream.writeBytes(buffer,(int) (position-start));
>> }
>>
>> SegmentReader.java
>> public int documents(byte[] buffer,int[] lengths,int start,int
>> max) throws IOException {
>> return fieldsReader.docs(buffer,lengths,start,max);
>> }
>>
>> SegmentMerger.java
>> private final int mergeFields() throws IOException {
>> fieldInfos = new FieldInfos(); // merge field names
>> int docCount = 0;
>> for (int i = 0; i < readers.size(); i++) {
>> IndexReader reader = (IndexReader) readers.elementAt(i);
>> if (reader instanceof SegmentReader) {
>> SegmentReader sreader = (SegmentReader) reader;
>> for (int j = 0; j < sreader.getFieldInfos().size(); j++) {
>> FieldInfo fi = sreader.getFieldInfos().fieldInfo(j);
>> fieldInfos.add(fi.name, fi.isIndexed, fi.storeTermVector,
>> fi.storePositionWithTermVector, fi.storeOffsetWithTermVector, !
>> reader.hasNorms(fi.name));
>> }
>> } else {
>> addIndexed(reader, fieldInfos, reader.getFieldNames
>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION_OFFSET), true,
>> true, true);
>> addIndexed(reader, fieldInfos, reader.getFieldNames
>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION), true, true,
>> false);
>> addIndexed(reader, fieldInfos, reader.getFieldNames
>> (IndexReader.FieldOption.TERMVECTOR_WITH_OFFSET), true, false, true);
>> addIndexed(reader, fieldInfos, reader.getFieldNames
>> (IndexReader.FieldOption.TERMVECTOR), true, false, false);
>> addIndexed(reader, fieldInfos, reader.getFieldNames
>> (IndexReader.FieldOption.INDEXED), false, false, false);
>> fieldInfos.add(reader.getFieldNames
>> (IndexReader.FieldOption.UNINDEXED), false);
>> }
>> }
>> fieldInfos.write(directory, segment + ".fnm");
>>
>> SegmentReader[] sreaders = new SegmentReader[readers.size()];
>> for (int i = 0; i < readers.size(); i++) {
>> IndexReader reader = (IndexReader) readers.elementAt(i);
>> boolean same = reader.getFieldNames().size() ==
>> fieldInfos.size() && reader instanceof SegmentReader;
>> if(same) {
>> SegmentReader sreader = (SegmentReader) reader;
>> for (int j = 0; same && j < fieldInfos.size(); j++) {
>> same = fieldInfos.fieldName(j).equals(sreader.getFieldInfos
>> ().fieldName(j));
>> }
>> if(same)
>> sreaders[i] = sreader;
>> }
>> }
>>
>> byte[] buffer = new byte[512*1024];
>> int[] lengths = new int[10000];
>>
>> // merge field values
>> FieldsWriter fieldsWriter = new FieldsWriter(directory, segment,
>> fieldInfos);
>>
>> try {
>> for (int i = 0; i < readers.size(); i++) {
>> IndexReader reader = (IndexReader) readers.elementAt(i);
>> SegmentReader sreader = sreaders[i];
>> int maxDoc = reader.maxDoc();
>> for (int j = 0; j < maxDoc;)
>> if (!reader.isDeleted(j)) { // skip deleted docs
>> if (sreader!=null) {
>> int start=j;
>> int ndocs=1;
>> for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j+
>> +,ndocs++);
>> ndocs = sreader.documents(buffer,lengths,start,j);
>> fieldsWriter.addDocuments(buffer,lengths,ndocs);
>> } else {
>> fieldsWriter.addDocument(reader.document(j));
>> j++;
>> }
>> docCount++;
>> }
>> }
>> } finally {
>> fieldsWriter.close();
>> }
>> return docCount;
>> }
>>
>>
>>
>>
>> On Nov 2, 2007, at 3:49 PM, robert engels wrote:
>>
>>> I am working on a bulk document copy right now - I will let you
>>> know if it improves things much.
>>>
>>> I doubt it, because I already configure the streams to use fairly
>>> large input and output buffers during a merge - but the memory
>>> index merge may see additional benefits due to lower CPU calls.
>>>
>>>
>>> On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
>>>
>>>>
>>>> [ https://issues.apache.org/jira/browse/LUCENE-1043?
>>>> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>>>> tabpanel#action_12539696 ]
>>>>
>>>> robert engels commented on LUCENE-1043:
>>>> ---------------------------------------
>>>>
>>>> When bulk copying the documents, I think you need to:
>>>>
>>>> read array of long from index (8 * (ndocs+1)) in long[ndocs+1]
>>>> offsets;
>>>> calculate length = offset[ndocs]-offset[0];
>>>> read bytes of length from document file
>>>> startoffset = current output document stream position
>>>> write bytes to output document
>>>> modify offset[] adding startoffset-offset[0] to each entry
>>>> write offset[] in bulk to index output
>>>>
>>>>> Speedup merging of stored fields when field mapping "matches"
>>>>> -------------------------------------------------------------
>>>>>
>>>>> Key: LUCENE-1043
>>>>> URL: https://issues.apache.org/jira/browse/
>>>>> LUCENE-1043
>>>>> Project: Lucene - Java
>>>>> Issue Type: Improvement
>>>>> Components: Index
>>>>> Affects Versions: 2.2
>>>>> Reporter: Michael McCandless
>>>>> Assignee: Michael McCandless
>>>>> Priority: Minor
>>>>> Fix For: 2.3
>>>>>
>>>>> Attachments: LUCENE-1043.patch
>>>>>
>>>>>
>>>>> Robert Engels suggested the following idea, here:
>>>>> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>>>>> When merging in the stored fields from a segment, if the field
>>>>> name ->
>>>>> number mapping is identical then we can simply bulk copy the
>>>>> entire
>>>>> entry for the document rather than re-interpreting and then re-
>>>>> writing
>>>>> the actual stored fields.
>>>>> I've pulled the code from the above thread and got it working
>>>>> on the
>>>>> current trunk.
>>>>
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> -
>>>> You can reply to this email to add a comment to the issue online.
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>
Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Posted by robert engels <re...@ix.netcom.com>.
Oops. Be careful with that code, merging works but indexing fails...
I have a bug lurking...
On Nov 2, 2007, at 4:28 PM, robert engels wrote:
> As expected it did not make much difference. Here is the code:
>
> FieldsReader.java
> final int docs(byte[] buffer,int[] lengths,int start,int max)
> throws IOException {
> final IndexInput indexStream = (IndexInput) indexLocal.get();
> indexStream.seek(start * 8L);
> long startoffset = indexStream.readLong();
> long lastoffset = startoffset;
> int count=0;
> while(count<max) {
> long offset = indexStream.readLong();
> if(offset-startoffset>buffer.length)
> break;
> lengths[count]=(int) (offset-lastoffset);
> lastoffset = offset;
> count++;
> }
> if(count==0)
> throw new IllegalStateException("buffer too small for a single
> document");
>
> final IndexInput fieldsStream = (IndexInput) fieldsLocal.get();
> fieldsStream.seek(startoffset);
> fieldsStream.readBytes(buffer,0,(int) (lastoffset-startoffset));
>
> return count;
> }
>
> FieldsWriter.java
> final void addDocuments(byte[] buffer, int[] lengths, int
> ndocs) throws IOException {
> long position = fieldsStream.getFilePointer();
> long start = position;
> for(int i=0;i<ndocs;i++) {
> indexStream.writeLong(position);
> position+=lengths[i];
> }
> fieldsStream.writeBytes(buffer,(int) (position-start));
> }
>
> SegmentReader.java
> public int documents(byte[] buffer,int[] lengths,int start,int
> max) throws IOException {
> return fieldsReader.docs(buffer,lengths,start,max);
> }
>
> SegmentMerger.java
> private final int mergeFields() throws IOException {
> fieldInfos = new FieldInfos(); // merge field names
> int docCount = 0;
> for (int i = 0; i < readers.size(); i++) {
> IndexReader reader = (IndexReader) readers.elementAt(i);
> if (reader instanceof SegmentReader) {
> SegmentReader sreader = (SegmentReader) reader;
> for (int j = 0; j < sreader.getFieldInfos().size(); j++) {
> FieldInfo fi = sreader.getFieldInfos().fieldInfo(j);
> fieldInfos.add(fi.name, fi.isIndexed, fi.storeTermVector,
> fi.storePositionWithTermVector, fi.storeOffsetWithTermVector, !
> reader.hasNorms(fi.name));
> }
> } else {
> addIndexed(reader, fieldInfos, reader.getFieldNames
> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION_OFFSET), true,
> true, true);
> addIndexed(reader, fieldInfos, reader.getFieldNames
> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION), true, true,
> false);
> addIndexed(reader, fieldInfos, reader.getFieldNames
> (IndexReader.FieldOption.TERMVECTOR_WITH_OFFSET), true, false, true);
> addIndexed(reader, fieldInfos, reader.getFieldNames
> (IndexReader.FieldOption.TERMVECTOR), true, false, false);
> addIndexed(reader, fieldInfos, reader.getFieldNames
> (IndexReader.FieldOption.INDEXED), false, false, false);
> fieldInfos.add(reader.getFieldNames
> (IndexReader.FieldOption.UNINDEXED), false);
> }
> }
> fieldInfos.write(directory, segment + ".fnm");
>
> SegmentReader[] sreaders = new SegmentReader[readers.size()];
> for (int i = 0; i < readers.size(); i++) {
> IndexReader reader = (IndexReader) readers.elementAt(i);
> boolean same = reader.getFieldNames().size() == fieldInfos.size
> () && reader instanceof SegmentReader;
> if(same) {
> SegmentReader sreader = (SegmentReader) reader;
> for (int j = 0; same && j < fieldInfos.size(); j++) {
> same = fieldInfos.fieldName(j).equals(sreader.getFieldInfos
> ().fieldName(j));
> }
> if(same)
> sreaders[i] = sreader;
> }
> }
>
> byte[] buffer = new byte[512*1024];
> int[] lengths = new int[10000];
>
> // merge field values
> FieldsWriter fieldsWriter = new FieldsWriter(directory, segment,
> fieldInfos);
>
> try {
> for (int i = 0; i < readers.size(); i++) {
> IndexReader reader = (IndexReader) readers.elementAt(i);
> SegmentReader sreader = sreaders[i];
> int maxDoc = reader.maxDoc();
> for (int j = 0; j < maxDoc;)
> if (!reader.isDeleted(j)) { // skip deleted docs
> if (sreader!=null) {
> int start=j;
> int ndocs=1;
> for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j+
> +,ndocs++);
> ndocs = sreader.documents(buffer,lengths,start,j);
> fieldsWriter.addDocuments(buffer,lengths,ndocs);
> } else {
> fieldsWriter.addDocument(reader.document(j));
> j++;
> }
> docCount++;
> }
> }
> } finally {
> fieldsWriter.close();
> }
> return docCount;
> }
>
>
>
>
> On Nov 2, 2007, at 3:49 PM, robert engels wrote:
>
>> I am working on a bulk document copy right now - I will let you
>> know if it improves things much.
>>
>> I doubt it, because I already configure the streams to use fairly
>> large input and output buffers during a merge - but the memory
>> index merge may see additional benefits due to lower CPU calls.
>>
>>
>> On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
>>
>>>
>>> [ https://issues.apache.org/jira/browse/LUCENE-1043?
>>> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>>> tabpanel#action_12539696 ]
>>>
>>> robert engels commented on LUCENE-1043:
>>> ---------------------------------------
>>>
>>> When bulk copying the documents, I think you need to:
>>>
>>> read array of long from index (8 * (ndocs+1)) in long[ndocs+1]
>>> offsets;
>>> calculate length = offset[ndocs]-offset[0];
>>> read bytes of length from document file
>>> startoffset = current output document stream position
>>> write bytes to output document
>>> modify offset[] adding startoffset-offset[0] to each entry
>>> write offset[] in bulk to index output
>>>
>>>> Speedup merging of stored fields when field mapping "matches"
>>>> -------------------------------------------------------------
>>>>
>>>> Key: LUCENE-1043
>>>> URL: https://issues.apache.org/jira/browse/
>>>> LUCENE-1043
>>>> Project: Lucene - Java
>>>> Issue Type: Improvement
>>>> Components: Index
>>>> Affects Versions: 2.2
>>>> Reporter: Michael McCandless
>>>> Assignee: Michael McCandless
>>>> Priority: Minor
>>>> Fix For: 2.3
>>>>
>>>> Attachments: LUCENE-1043.patch
>>>>
>>>>
>>>> Robert Engels suggested the following idea, here:
>>>> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>>>> When merging in the stored fields from a segment, if the field
>>>> name ->
>>>> number mapping is identical then we can simply bulk copy the entire
>>>> entry for the document rather than re-interpreting and then re-
>>>> writing
>>>> the actual stored fields.
>>>> I've pulled the code from the above thread and got it working on
>>>> the
>>>> current trunk.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> You can reply to this email to add a comment to the issue online.
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Posted by Yonik Seeley <yo...@apache.org>.
On 11/2/07, robert engels <re...@ix.netcom.com> wrote:
> As expected it did not make much difference. Here is the code:
Yeah, might already be IO bound with FSindex.
Might see better improvement with a RAMDir since copying in bigger
chunks eliminates the double buffering so you go from 4 copies
OS -> IndexInput.buffer -> user.buffer -> IndexOutput.buffer -> OS
to 2 copies
OS -> user.buffer -> OS
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Posted by robert engels <re...@ix.netcom.com>.
As expected it did not make much difference. Here is the code:
FieldsReader.java
final int docs(byte[] buffer,int[] lengths,int start,int max)
throws IOException {
final IndexInput indexStream = (IndexInput) indexLocal.get();
indexStream.seek(start * 8L);
long startoffset = indexStream.readLong();
long lastoffset = startoffset;
int count=0;
while(count<max) {
long offset = indexStream.readLong();
if(offset-startoffset>buffer.length)
break;
lengths[count]=(int) (offset-lastoffset);
lastoffset = offset;
count++;
}
if(count==0)
throw new IllegalStateException("buffer too small for a single
document");
final IndexInput fieldsStream = (IndexInput) fieldsLocal.get();
fieldsStream.seek(startoffset);
fieldsStream.readBytes(buffer,0,(int) (lastoffset-startoffset));
return count;
}
FieldsWriter.java
final void addDocuments(byte[] buffer, int[] lengths, int ndocs)
throws IOException {
long position = fieldsStream.getFilePointer();
long start = position;
for(int i=0;i<ndocs;i++) {
indexStream.writeLong(position);
position+=lengths[i];
}
fieldsStream.writeBytes(buffer,(int) (position-start));
}
SegmentReader.java
public int documents(byte[] buffer,int[] lengths,int start,int
max) throws IOException {
return fieldsReader.docs(buffer,lengths,start,max);
}
SegmentMerger.java
private final int mergeFields() throws IOException {
fieldInfos = new FieldInfos(); // merge field names
int docCount = 0;
for (int i = 0; i < readers.size(); i++) {
IndexReader reader = (IndexReader) readers.elementAt(i);
if (reader instanceof SegmentReader) {
SegmentReader sreader = (SegmentReader) reader;
for (int j = 0; j < sreader.getFieldInfos().size(); j++) {
FieldInfo fi = sreader.getFieldInfos().fieldInfo(j);
fieldInfos.add(fi.name, fi.isIndexed, fi.storeTermVector,
fi.storePositionWithTermVector, fi.storeOffsetWithTermVector, !
reader.hasNorms(fi.name));
}
} else {
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.TERMVECTOR_WITH_POSITION_OFFSET), true,
true, true);
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.TERMVECTOR_WITH_POSITION), true, true, false);
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.TERMVECTOR_WITH_OFFSET), true, false, true);
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.TERMVECTOR), true, false, false);
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.INDEXED), false, false, false);
fieldInfos.add(reader.getFieldNames
(IndexReader.FieldOption.UNINDEXED), false);
}
}
fieldInfos.write(directory, segment + ".fnm");
SegmentReader[] sreaders = new SegmentReader[readers.size()];
for (int i = 0; i < readers.size(); i++) {
IndexReader reader = (IndexReader) readers.elementAt(i);
boolean same = reader.getFieldNames().size() == fieldInfos.size
() && reader instanceof SegmentReader;
if(same) {
SegmentReader sreader = (SegmentReader) reader;
for (int j = 0; same && j < fieldInfos.size(); j++) {
same = fieldInfos.fieldName(j).equals(sreader.getFieldInfos
().fieldName(j));
}
if(same)
sreaders[i] = sreader;
}
}
byte[] buffer = new byte[512*1024];
int[] lengths = new int[10000];
// merge field values
FieldsWriter fieldsWriter = new FieldsWriter(directory, segment,
fieldInfos);
try {
for (int i = 0; i < readers.size(); i++) {
IndexReader reader = (IndexReader) readers.elementAt(i);
SegmentReader sreader = sreaders[i];
int maxDoc = reader.maxDoc();
for (int j = 0; j < maxDoc;)
if (!reader.isDeleted(j)) { // skip deleted docs
if (sreader!=null) {
int start=j;
int ndocs=1;
for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j+
+,ndocs++);
ndocs = sreader.documents(buffer,lengths,start,j);
fieldsWriter.addDocuments(buffer,lengths,ndocs);
} else {
fieldsWriter.addDocument(reader.document(j));
j++;
}
docCount++;
}
}
} finally {
fieldsWriter.close();
}
return docCount;
}
On Nov 2, 2007, at 3:49 PM, robert engels wrote:
> I am working on a bulk document copy right now - I will let you
> know if it improves things much.
>
> I doubt it, because I already configure the streams to use fairly
> large input and output buffers during a merge - but the memory
> index merge may see additional benefits due to lower CPU calls.
>
>
> On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
>
>>
>> [ https://issues.apache.org/jira/browse/LUCENE-1043?
>> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel#action_12539696 ]
>>
>> robert engels commented on LUCENE-1043:
>> ---------------------------------------
>>
>> When bulk copying the documents, I think you need to:
>>
>> read array of long from index (8 * (ndocs+1)) in long[ndocs+1]
>> offsets;
>> calculate length = offset[ndocs]-offset[0];
>> read bytes of length from document file
>> startoffset = current output document stream position
>> write bytes to output document
>> modify offset[] adding startoffset-offset[0] to each entry
>> write offset[] in bulk to index output
>>
>>> Speedup merging of stored fields when field mapping "matches"
>>> -------------------------------------------------------------
>>>
>>> Key: LUCENE-1043
>>> URL: https://issues.apache.org/jira/browse/
>>> LUCENE-1043
>>> Project: Lucene - Java
>>> Issue Type: Improvement
>>> Components: Index
>>> Affects Versions: 2.2
>>> Reporter: Michael McCandless
>>> Assignee: Michael McCandless
>>> Priority: Minor
>>> Fix For: 2.3
>>>
>>> Attachments: LUCENE-1043.patch
>>>
>>>
>>> Robert Engels suggested the following idea, here:
>>> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>>> When merging in the stored fields from a segment, if the field
>>> name ->
>>> number mapping is identical then we can simply bulk copy the entire
>>> entry for the document rather than re-interpreting and then re-
>>> writing
>>> the actual stored fields.
>>> I've pulled the code from the above thread and got it working on the
>>> current trunk.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Posted by robert engels <re...@ix.netcom.com>.
I am working on a bulk document copy right now - I will let you know
if it improves things much.
I doubt it, because I already configure the streams to use fairly
large input and output buffers during a merge - but the memory index
merge may see additional benefits due to lower CPU calls.
On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENE-1043?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel#action_12539696 ]
>
> robert engels commented on LUCENE-1043:
> ---------------------------------------
>
> When bulk copying the documents, I think you need to:
>
> read array of long from index (8 * (ndocs+1)) in long[ndocs+1]
> offsets;
> calculate length = offset[ndocs]-offset[0];
> read bytes of length from document file
> startoffset = current output document stream position
> write bytes to output document
> modify offset[] adding startoffset-offset[0] to each entry
> write offset[] in bulk to index output
>
>> Speedup merging of stored fields when field mapping "matches"
>> -------------------------------------------------------------
>>
>> Key: LUCENE-1043
>> URL: https://issues.apache.org/jira/browse/
>> LUCENE-1043
>> Project: Lucene - Java
>> Issue Type: Improvement
>> Components: Index
>> Affects Versions: 2.2
>> Reporter: Michael McCandless
>> Assignee: Michael McCandless
>> Priority: Minor
>> Fix For: 2.3
>>
>> Attachments: LUCENE-1043.patch
>>
>>
>> Robert Engels suggested the following idea, here:
>> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>> When merging in the stored fields from a segment, if the field
>> name ->
>> number mapping is identical then we can simply bulk copy the entire
>> entry for the document rather than re-interpreting and then re-
>> writing
>> the actual stored fields.
>> I've pulled the code from the above thread and got it working on the
>> current trunk.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1043) Speedup merging of stored fields
when field mapping "matches"
Posted by "robert engels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539696 ]
robert engels commented on LUCENE-1043:
---------------------------------------
When bulk copying the documents, I think you need to:
read array of long from index (8 * (ndocs+1)) in long[ndocs+1] offsets;
calculate length = offset[ndocs]-offset[0];
read bytes of length from document file
startoffset = current output document stream position
write bytes to output document
modify offset[] adding startoffset-offset[0] to each entry
write offset[] in bulk to index output
> Speedup merging of stored fields when field mapping "matches"
> -------------------------------------------------------------
>
> Key: LUCENE-1043
> URL: https://issues.apache.org/jira/browse/LUCENE-1043
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1043.patch
>
>
> Robert Engels suggested the following idea, here:
> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
> When merging in the stored fields from a segment, if the field name ->
> number mapping is identical then we can simply bulk copy the entire
> entry for the document rather than re-interpreting and then re-writing
> the actual stored fields.
> I've pulled the code from the above thread and got it working on the
> current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1043) Speedup merging of stored fields when
field mapping "matches"
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1043:
---------------------------------------
Attachment: LUCENE-1043.patch
Initial patch. All tests pass.
> Speedup merging of stored fields when field mapping "matches"
> -------------------------------------------------------------
>
> Key: LUCENE-1043
> URL: https://issues.apache.org/jira/browse/LUCENE-1043
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1043.patch
>
>
> Robert Engels suggested the following idea, here:
> http://www.gossamer-threads.com/lists/lucene/java-dev/54217
> When merging in the stored fields from a segment, if the field name ->
> number mapping is identical then we can simply bulk copy the entire
> entry for the document rather than re-interpreting and then re-writing
> the actual stored fields.
> I've pulled the code from the above thread and got it working on the
> current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org