You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/09/09 08:30:26 UTC
[GitHub] [lucene-solr] s1monw opened a new pull request #1848: LUCENE-9515: Detach DWPT from DefaultIndexingChain
s1monw opened a new pull request #1848:
URL: https://github.com/apache/lucene-solr/pull/1848
This change removes the DWPT dependency from DefaultIndexingChain
and rather passes on the primitives needed for creating the chain.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1848: LUCENE-9515: Detach DWPT from DefaultIndexingChain
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1848:
URL: https://github.com/apache/lucene-solr/pull/1848#discussion_r485499853
##########
File path: lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java
##########
@@ -73,21 +78,38 @@
// Holds fields seen in each document
private PerField[] fields = new PerField[1];
private final InfoStream infoStream;
-
- DefaultIndexingChain(DocumentsWriterPerThread docWriter) {
- this.docWriter = docWriter;
- this.fieldInfos = docWriter.getFieldInfosBuilder();
- this.infoStream = docWriter.getIndexWriterConfig().getInfoStream();
+ private final ByteBlockPool.Allocator byteBlockAllocator;
+ private final LiveIndexWriterConfig indexWriterConfig;
+ private final int indexCreatedVersionMajor;
+ private final Consumer<Throwable> abortingExceptionConsumer;
+ private boolean hasHitAbortingException = false;
+
+ DefaultIndexingChain(int indexCreatedVersionMajor, SegmentInfo segmentInfo, Directory directory, FieldInfos.Builder fieldInfos, LiveIndexWriterConfig indexWriterConfig,
+ Consumer<Throwable> abortingExceptionConsumer) {
+ this.indexCreatedVersionMajor = indexCreatedVersionMajor;
+ byteBlockAllocator = new ByteBlockPool.DirectTrackingAllocator(bytesUsed);
+ IntBlockPool.Allocator intBlockAllocator = new IntBlockAllocator(bytesUsed);
+ this.indexWriterConfig = indexWriterConfig;
+ assert segmentInfo.getIndexSort() == indexWriterConfig.getIndexSort();
+ this.fieldInfos = fieldInfos;
+ this.infoStream = indexWriterConfig.getInfoStream();
+ this.abortingExceptionConsumer = abortingExceptionConsumer;
final TermsHash termVectorsWriter;
- if (docWriter.getSegmentInfo().getIndexSort() == null) {
- storedFieldsConsumer = new StoredFieldsConsumer(docWriter.codec, docWriter.directory, docWriter.getSegmentInfo());
- termVectorsWriter = new TermVectorsConsumer(docWriter);
+ if (segmentInfo.getIndexSort() == null) {
+ storedFieldsConsumer = new StoredFieldsConsumer(indexWriterConfig.getCodec(), directory, segmentInfo);
+ termVectorsWriter = new TermVectorsConsumer(intBlockAllocator, byteBlockAllocator, directory, segmentInfo, indexWriterConfig.getCodec());
} else {
- storedFieldsConsumer = new SortingStoredFieldsConsumer(docWriter.codec, docWriter.directory, docWriter.getSegmentInfo());
- termVectorsWriter = new SortingTermVectorsConsumer(docWriter);
+ storedFieldsConsumer = new SortingStoredFieldsConsumer(indexWriterConfig.getCodec(), directory, segmentInfo);
+ termVectorsWriter = new SortingTermVectorsConsumer(intBlockAllocator, byteBlockAllocator, directory, segmentInfo, indexWriterConfig.getCodec());
}
- termsHash = new FreqProxTermsWriter(docWriter, bytesUsed, termVectorsWriter);
+ termsHash = new FreqProxTermsWriter(intBlockAllocator, byteBlockAllocator, bytesUsed, termVectorsWriter);
+ }
+
+ private void onAbortingException(Throwable th) {
Review comment:
This was moved from `DWPT`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1848: LUCENE-9515: Detach DWPT from DefaultIndexingChain
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1848:
URL: https://github.com/apache/lucene-solr/pull/1848#discussion_r485499853
##########
File path: lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java
##########
@@ -73,21 +78,38 @@
// Holds fields seen in each document
private PerField[] fields = new PerField[1];
private final InfoStream infoStream;
-
- DefaultIndexingChain(DocumentsWriterPerThread docWriter) {
- this.docWriter = docWriter;
- this.fieldInfos = docWriter.getFieldInfosBuilder();
- this.infoStream = docWriter.getIndexWriterConfig().getInfoStream();
+ private final ByteBlockPool.Allocator byteBlockAllocator;
+ private final LiveIndexWriterConfig indexWriterConfig;
+ private final int indexCreatedVersionMajor;
+ private final Consumer<Throwable> abortingExceptionConsumer;
+ private boolean hasHitAbortingException = false;
+
+ DefaultIndexingChain(int indexCreatedVersionMajor, SegmentInfo segmentInfo, Directory directory, FieldInfos.Builder fieldInfos, LiveIndexWriterConfig indexWriterConfig,
+ Consumer<Throwable> abortingExceptionConsumer) {
+ this.indexCreatedVersionMajor = indexCreatedVersionMajor;
+ byteBlockAllocator = new ByteBlockPool.DirectTrackingAllocator(bytesUsed);
+ IntBlockPool.Allocator intBlockAllocator = new IntBlockAllocator(bytesUsed);
+ this.indexWriterConfig = indexWriterConfig;
+ assert segmentInfo.getIndexSort() == indexWriterConfig.getIndexSort();
+ this.fieldInfos = fieldInfos;
+ this.infoStream = indexWriterConfig.getInfoStream();
+ this.abortingExceptionConsumer = abortingExceptionConsumer;
final TermsHash termVectorsWriter;
- if (docWriter.getSegmentInfo().getIndexSort() == null) {
- storedFieldsConsumer = new StoredFieldsConsumer(docWriter.codec, docWriter.directory, docWriter.getSegmentInfo());
- termVectorsWriter = new TermVectorsConsumer(docWriter);
+ if (segmentInfo.getIndexSort() == null) {
+ storedFieldsConsumer = new StoredFieldsConsumer(indexWriterConfig.getCodec(), directory, segmentInfo);
+ termVectorsWriter = new TermVectorsConsumer(intBlockAllocator, byteBlockAllocator, directory, segmentInfo, indexWriterConfig.getCodec());
} else {
- storedFieldsConsumer = new SortingStoredFieldsConsumer(docWriter.codec, docWriter.directory, docWriter.getSegmentInfo());
- termVectorsWriter = new SortingTermVectorsConsumer(docWriter);
+ storedFieldsConsumer = new SortingStoredFieldsConsumer(indexWriterConfig.getCodec(), directory, segmentInfo);
+ termVectorsWriter = new SortingTermVectorsConsumer(intBlockAllocator, byteBlockAllocator, directory, segmentInfo, indexWriterConfig.getCodec());
}
- termsHash = new FreqProxTermsWriter(docWriter, bytesUsed, termVectorsWriter);
+ termsHash = new FreqProxTermsWriter(intBlockAllocator, byteBlockAllocator, bytesUsed, termVectorsWriter);
+ }
+
+ private void onAbortingException(Throwable th) {
Review comment:
This was moved from `DWPT`?
Ahh, no, we forked it from there, ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1848: LUCENE-9515: Detach DWPT from DefaultIndexingChain
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1848:
URL: https://github.com/apache/lucene-solr/pull/1848#discussion_r485644701
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -272,7 +273,9 @@ static int getActualMaxDocs() {
* and a message is printed to infoStream, if set (see {@link
* IndexWriterConfig#setInfoStream(InfoStream)}).
*/
- public final static int MAX_TERM_LENGTH = DocumentsWriterPerThread.MAX_TERM_LENGTH_UTF8;
+ // if you increase this, you must fix field cache impl for
+ // getTerms/getTermsIndex requires <= 32768
Review comment:
Maybe remove this stale comment :) FieldCache is long gone!
##########
File path: lucene/core/src/test/org/apache/lucene/index/TestIndexWriter.java
##########
@@ -1533,7 +1533,7 @@ public void testWickedLongTerm() throws IOException {
Directory dir = newDirectory();
RandomIndexWriter w = new RandomIndexWriter(random(), dir, new StringSplitAnalyzer());
- char[] chars = new char[DocumentsWriterPerThread.MAX_TERM_LENGTH_UTF8];
+ char[] chars = new char[IndexWriter.MAX_TERM_LENGTH];
Review comment:
Ooh good catch (`char[]` vs `byte[]`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1848: LUCENE-9515: Detach DWPT from DefaultIndexingChain
Posted by GitBox <gi...@apache.org>.
s1monw commented on a change in pull request #1848:
URL: https://github.com/apache/lucene-solr/pull/1848#discussion_r487697084
##########
File path: lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java
##########
@@ -73,21 +78,38 @@
// Holds fields seen in each document
private PerField[] fields = new PerField[1];
private final InfoStream infoStream;
-
- DefaultIndexingChain(DocumentsWriterPerThread docWriter) {
- this.docWriter = docWriter;
- this.fieldInfos = docWriter.getFieldInfosBuilder();
- this.infoStream = docWriter.getIndexWriterConfig().getInfoStream();
+ private final ByteBlockPool.Allocator byteBlockAllocator;
+ private final LiveIndexWriterConfig indexWriterConfig;
+ private final int indexCreatedVersionMajor;
+ private final Consumer<Throwable> abortingExceptionConsumer;
+ private boolean hasHitAbortingException = false;
+
+ DefaultIndexingChain(int indexCreatedVersionMajor, SegmentInfo segmentInfo, Directory directory, FieldInfos.Builder fieldInfos, LiveIndexWriterConfig indexWriterConfig,
+ Consumer<Throwable> abortingExceptionConsumer) {
+ this.indexCreatedVersionMajor = indexCreatedVersionMajor;
+ byteBlockAllocator = new ByteBlockPool.DirectTrackingAllocator(bytesUsed);
+ IntBlockPool.Allocator intBlockAllocator = new IntBlockAllocator(bytesUsed);
+ this.indexWriterConfig = indexWriterConfig;
+ assert segmentInfo.getIndexSort() == indexWriterConfig.getIndexSort();
+ this.fieldInfos = fieldInfos;
+ this.infoStream = indexWriterConfig.getInfoStream();
+ this.abortingExceptionConsumer = abortingExceptionConsumer;
final TermsHash termVectorsWriter;
- if (docWriter.getSegmentInfo().getIndexSort() == null) {
- storedFieldsConsumer = new StoredFieldsConsumer(docWriter.codec, docWriter.directory, docWriter.getSegmentInfo());
- termVectorsWriter = new TermVectorsConsumer(docWriter);
+ if (segmentInfo.getIndexSort() == null) {
+ storedFieldsConsumer = new StoredFieldsConsumer(indexWriterConfig.getCodec(), directory, segmentInfo);
+ termVectorsWriter = new TermVectorsConsumer(intBlockAllocator, byteBlockAllocator, directory, segmentInfo, indexWriterConfig.getCodec());
} else {
- storedFieldsConsumer = new SortingStoredFieldsConsumer(docWriter.codec, docWriter.directory, docWriter.getSegmentInfo());
- termVectorsWriter = new SortingTermVectorsConsumer(docWriter);
+ storedFieldsConsumer = new SortingStoredFieldsConsumer(indexWriterConfig.getCodec(), directory, segmentInfo);
+ termVectorsWriter = new SortingTermVectorsConsumer(intBlockAllocator, byteBlockAllocator, directory, segmentInfo, indexWriterConfig.getCodec());
}
- termsHash = new FreqProxTermsWriter(docWriter, bytesUsed, termVectorsWriter);
+ termsHash = new FreqProxTermsWriter(intBlockAllocator, byteBlockAllocator, bytesUsed, termVectorsWriter);
+ }
+
+ private void onAbortingException(Throwable th) {
Review comment:
yeah I just tried to not repeat the assignment of `hasHitAbortingException`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw merged pull request #1848: LUCENE-9515: Detach DWPT from DefaultIndexingChain
Posted by GitBox <gi...@apache.org>.
s1monw merged pull request #1848:
URL: https://github.com/apache/lucene-solr/pull/1848
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1848: LUCENE-9515: Detach DWPT from DefaultIndexingChain
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1848:
URL: https://github.com/apache/lucene-solr/pull/1848#discussion_r485498630
##########
File path: lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java
##########
@@ -73,21 +78,38 @@
// Holds fields seen in each document
private PerField[] fields = new PerField[1];
private final InfoStream infoStream;
-
- DefaultIndexingChain(DocumentsWriterPerThread docWriter) {
- this.docWriter = docWriter;
- this.fieldInfos = docWriter.getFieldInfosBuilder();
- this.infoStream = docWriter.getIndexWriterConfig().getInfoStream();
+ private final ByteBlockPool.Allocator byteBlockAllocator;
+ private final LiveIndexWriterConfig indexWriterConfig;
+ private final int indexCreatedVersionMajor;
+ private final Consumer<Throwable> abortingExceptionConsumer;
+ private boolean hasHitAbortingException = false;
Review comment:
Don't need the `= false` -- it is java's default already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org