You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by jp...@apache.org on 2023/01/11 09:03:10 UTC
[lucene] branch branch_9x updated (eac4da8eb01 -> b8d40e6af8d)
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a change to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
from eac4da8eb01 Documenting that IndexReaderContext#leaves() will never return a null value and remove the null checks from the method calls (#12034)
new 87d8ab9ca28 Cut over Lucene Demo from LongPoint to LongField. (#12052)
new b8d40e6af8d Never throttle creation of compound files. (#12070)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
lucene/CHANGES.txt | 3 +++
lucene/core/src/java/org/apache/lucene/index/IndexWriter.java | 5 ++++-
lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java | 9 +++++----
3 files changed, 12 insertions(+), 5 deletions(-)
[lucene] 01/02: Cut over Lucene Demo from LongPoint to LongField. (#12052)
Posted by jp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 87d8ab9ca282ddebf88546bf29e653302846a0e2
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Wed Jan 11 09:43:43 2023 +0100
Cut over Lucene Demo from LongPoint to LongField. (#12052)
---
lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java b/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java
index defadfb7d23..adcf824b52e 100644
--- a/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java
+++ b/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java
@@ -35,7 +35,7 @@ import org.apache.lucene.demo.knn.KnnVectorDict;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.KnnVectorField;
-import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.document.LongField;
import org.apache.lucene.document.StringField;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
@@ -238,13 +238,14 @@ public class IndexFiles implements AutoCloseable {
doc.add(pathField);
// Add the last modified date of the file a field named "modified".
- // Use a LongPoint that is indexed (i.e. efficiently filterable with
- // PointRangeQuery). This indexes to milli-second resolution, which
+ // Use a LongField that is indexed with points and doc values, and is efficient
+ // for both filtering (LongField#newRangeQuery) and sorting
+ // (LongField#newSortField). This indexes to milli-second resolution, which
// is often too fine. You could instead create a number based on
// year/month/day/hour/minutes/seconds, down the resolution you require.
// For example the long value 2011021714 would mean
// February 17, 2011, 2-3 PM.
- doc.add(new LongPoint("modified", lastModified));
+ doc.add(new LongField("modified", lastModified));
// Add the contents of the file to a field named "contents". Specify a Reader,
// so that the text of the file is tokenized and indexed, but not stored.
[lucene] 02/02: Never throttle creation of compound files. (#12070)
Posted by jp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit b8d40e6af8d71ac776fe743a8dbd7d68eed07846
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Wed Jan 11 09:57:13 2023 +0100
Never throttle creation of compound files. (#12070)
`ConcurrentMergeScheduler` uses the rate at which a merge writes bytes as a
proxy for CPU usage, in order to prevent merging from disrupting searches too
much. However creating compound files are lightweight CPU-wise and do not need
throttling.
Closes #12068
---
lucene/CHANGES.txt | 3 +++
lucene/core/src/java/org/apache/lucene/index/IndexWriter.java | 5 ++++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index d1802e42dea..eb9b5cf5403 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -105,6 +105,9 @@ Improvements
* GITHUB#12034: Remove null check in IndexReaderContext#leaves() usages (Erik Pellizzon)
+* GITHUB#12070: Compound file creation is no longer subject to merge throttling.
+ (Adrien Grand)
+
Bug Fixes
---------------------
* GITHUB#11726: Indexing term vectors on large documents could fail due to
diff --git a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
index 88ed488dbad..f488e1f83f6 100644
--- a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
@@ -5211,7 +5211,10 @@ public class IndexWriter
success = false;
Collection<String> filesToRemove = merge.info.files();
- TrackingDirectoryWrapper trackingCFSDir = new TrackingDirectoryWrapper(mergeDirectory);
+ // NOTE: Creation of the CFS file must be performed with the original
+ // directory rather than with the merging directory, so that it is not
+ // subject to merge throttling.
+ TrackingDirectoryWrapper trackingCFSDir = new TrackingDirectoryWrapper(directory);
try {
createCompoundFile(
infoStream, trackingCFSDir, merge.info.info, context, this::deleteNewFiles);