You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by GitBox <gi...@apache.org> on 2021/08/06 17:20:04 UTC
[GitHub] [lucenenet] fjam opened a new issue #505: New .cfs files created everytime I add to index
fjam opened a new issue #505:
URL: https://github.com/apache/lucenenet/issues/505
On initial load of my application I add data into my Lucene index:
```
using (StandardAnalyzer analyzer = new StandardAnalyzer(LuceneVersion.LUCENE_48))
{
using (IndexWriter writer = new IndexWriter(Directory)
{
Document doc = new Document();
doc.Add(new TextField("Id", Id.ToString(), Lucene.Net.Documents.Field.Store.YES))
doc.Add(new TextField("Symbol", Symbol, Lucene.Net.Documents.Field.Store.NO))
doc.Add(new TextField("Channel", Channel, Lucene.Net.Documents.Field.Store.NO))
writer.AddDocument(doc);
writer.Commmit();
}
}
```
This creates a few files (_0.cfe, _0.cfs ...) in the index directory.
Now when I add need to add a new index to Lucene with the same Model (Id, Symbols, Channel) it creates a new set of files (_1.cfe, _1.cfs ...) in the index directory without removing the _0.cfs... files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo commented on issue #505: New .cfs files created everytime I add to index
Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #505:
URL: https://github.com/apache/lucenenet/issues/505#issuecomment-894533284
Inserting 1000s of docs is no problem. On my local my dev box I can insert around 14K docs per second. Of course the speed will depend greatly on the size of the documents and the speed of the machine.
Lucene manages the segments well and will merge them periodically. The result will be less segments but bigger segments. You can make Lucene's life easier and your indexing faster by not calling `Flush`, `Commit` or `MaybeRefreshBlocking` until after you have inserted the set of documents you desire. (These commands force buffered documents to be written out in a new segment). But even if you call all these methods after each time you add a document to the index Lucene will still manage the segments well by doing merging as needed. It will just be much slower (maybe like 20 docs per second).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo edited a comment on issue #505: New .cfs files created everytime I add to index
Posted by GitBox <gi...@apache.org>.
rclabo edited a comment on issue #505:
URL: https://github.com/apache/lucenenet/issues/505#issuecomment-894422703
Yep, that's the way Lucene.NET and Lucene works. Each of these sets of files is called a segment. Initially in your case there is one segment, the _0 segment, and then when you add another doc and commit there is a 2nd segment, the _1 segment. If you want to learn more google about lucene and segments. There is lots of info online. Example: https://stackoverflow.com/questions/2703432/what-are-segments-in-lucene
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] fjam commented on issue #505: New .cfs files created everytime I add to index
Posted by GitBox <gi...@apache.org>.
fjam commented on issue #505:
URL: https://github.com/apache/lucenenet/issues/505#issuecomment-894418876
I am using Lucene.NET version 4.8.0-beta00014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] rclabo commented on issue #505: New .cfs files created everytime I add to index
Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #505:
URL: https://github.com/apache/lucenenet/issues/505#issuecomment-894422703
Yep, that's the way Lucene.NET and Lucene works. Each of these sets of files is called a segment. Initially in your case there is one segment, the _0 segment, and then when you add another doc and commit there is a 2nd segment, the _1 segment. If you want to learn more google about lucene and segments. There is lots of info online.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] fjam commented on issue #505: New .cfs files created everytime I add to index
Posted by GitBox <gi...@apache.org>.
fjam commented on issue #505:
URL: https://github.com/apache/lucenenet/issues/505#issuecomment-894424982
I have a process running where there will be 1000s of documents being indexed a day (at different times). This way there will be 1000s of new .cfs files in my directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on issue #505: New .cfs files created everytime I add to index
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on issue #505:
URL: https://github.com/apache/lucenenet/issues/505#issuecomment-945181318
Closing as this question appears to be answered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 closed issue #505: New .cfs files created everytime I add to index
Posted by GitBox <gi...@apache.org>.
NightOwl888 closed issue #505:
URL: https://github.com/apache/lucenenet/issues/505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] fjam commented on issue #505: New .cfs files created everytime I add to index
Posted by GitBox <gi...@apache.org>.
fjam commented on issue #505:
URL: https://github.com/apache/lucenenet/issues/505#issuecomment-894425404
Is that correct way? or is there a way to merge these .cfs files together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org