You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/12/14 08:13:02 UTC

[GitHub] [lucene] ywelsch opened a new pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

ywelsch opened a new pull request #539:
URL: https://github.com/apache/lucene/pull/539


   # Description
   
   Unlike points, norms, term vectors or doc values which only get written to the directory when at least one of the fields uses the data structure, postings always get written to the directory.
   While this isn't hurting much, it can be surprising at times, e.g. if you index with SimpleText you will have a file for postings even though none of the fields indexes postings. This inconsistency is hidden with the default codec due to the fact that it uses PerFieldPostingsFormat, which only delegates to any of the per-field codecs if any of the fields is actually indexed, so you don't actually get a file if none of the fields is indexed.
   
   # Solution
   
   Fixes the situation by making read / write of postings conditional on whether there is any data indexed. This can be determined from the metadata in FieldInfos.
   
   # Tests
   
   Added new test `TestMinimalCodec`. Passed existing tests via ./gradlew clean; ./gradlew check
   
   # Checklist
   
   Please review the following and check all that apply:
   
   - [x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/lucene/HowToContribute) and my code conforms to the standards described there to the best of my ability.
   - [x] I have created a Jira issue and added the issue ID to my pull request title.
   - [x] I have given Lucene maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
   - [x] I have developed this patch against the `main` branch.
   - [x] I have run `./gradlew check`.
   - [x] I have added tests for my changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] jpountz commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r768421436



##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI

Review comment:
       should we register these 4 codecs via SPI?

##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI
+
+      IndexWriterConfig writerConfig =
+          newIndexWriterConfig(new MockAnalyzer(random()))
+              .setCodec(new MinimalCodec(useCompoundFile, useDeletes))
+              .setUseCompoundFile(useCompoundFile);
+      if (!useCompoundFile) {
+        writerConfig.getMergePolicy().setNoCFSRatio(0.0);
+        writerConfig.getMergePolicy().setMaxCFSSegmentSizeMB(Double.POSITIVE_INFINITY);
+      }
+
+      int expectedNumDocs = 0;
+      try (IndexWriter writer = new IndexWriter(dir, writerConfig)) {
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.flush(); // create second segment

Review comment:
       this comment is misplaced?

##########
File path: lucene/core/src/java/org/apache/lucene/index/SegmentMerger.java
##########
@@ -203,8 +203,13 @@ private void mergeTerms(SegmentWriteState segmentWriteState, SegmentReadState se
         // Use the merge instance in order to reuse the same IndexInput for all terms
         normsMergeInstance = norms.getMergeInstance();
       }
-      try (FieldsConsumer consumer = codec.postingsFormat().fieldsConsumer(segmentWriteState)) {
-        consumer.merge(mergeState, normsMergeInstance);
+      try (FieldsConsumer consumer =
+          mergeState.mergeFieldInfos.hasIndexed()
+              ? codec.postingsFormat().fieldsConsumer(segmentWriteState)
+              : null) {
+        if (consumer != null) {
+          consumer.merge(mergeState, normsMergeInstance);
+        }

Review comment:
       I think it would read better if we had the hasIndexed check outside of the try block, so that we would no longer need to check that `consumer` is not null?

##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI
+
+      IndexWriterConfig writerConfig =
+          newIndexWriterConfig(new MockAnalyzer(random()))
+              .setCodec(new MinimalCodec(useCompoundFile, useDeletes))
+              .setUseCompoundFile(useCompoundFile);
+      if (!useCompoundFile) {
+        writerConfig.getMergePolicy().setNoCFSRatio(0.0);
+        writerConfig.getMergePolicy().setMaxCFSSegmentSizeMB(Double.POSITIVE_INFINITY);
+      }
+
+      int expectedNumDocs = 0;
+      try (IndexWriter writer = new IndexWriter(dir, writerConfig)) {
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.flush(); // create second segment
+        if (useDeletes && randomBoolean()) {
+          writer.deleteDocuments(new MatchAllDocsQuery());
+          expectedNumDocs = 0;
+        }
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.forceMerge(1); // test merges
+        if (useDeletes && randomBoolean()) {
+          writer.deleteDocuments(new MatchAllDocsQuery());

Review comment:
       It looks like your test always deletes entire segments, which is a special case for Lucene. Can you only delete individual documents (and ideally less than 25% of the index to avoid triggering a merge immediately) to make sure live docs are correctly exercised?

##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI
+
+      IndexWriterConfig writerConfig =
+          newIndexWriterConfig(new MockAnalyzer(random()))
+              .setCodec(new MinimalCodec(useCompoundFile, useDeletes))
+              .setUseCompoundFile(useCompoundFile);
+      if (!useCompoundFile) {
+        writerConfig.getMergePolicy().setNoCFSRatio(0.0);
+        writerConfig.getMergePolicy().setMaxCFSSegmentSizeMB(Double.POSITIVE_INFINITY);
+      }
+
+      int expectedNumDocs = 0;
+      try (IndexWriter writer = new IndexWriter(dir, writerConfig)) {
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.flush(); // create second segment
+        if (useDeletes && randomBoolean()) {
+          writer.deleteDocuments(new MatchAllDocsQuery());
+          expectedNumDocs = 0;
+        }
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.forceMerge(1); // test merges
+        if (useDeletes && randomBoolean()) {
+          writer.deleteDocuments(new MatchAllDocsQuery());
+          expectedNumDocs = 0;
+        }
+        if (randomBoolean()) {
+          writer.commit();
+        }
+
+        try (DirectoryReader reader = DirectoryReader.open(writer)) {
+          assertEquals(expectedNumDocs, reader.numDocs());
+        }
+      }
+    }
+  }
+
+  /** returns a basic document with no indexed fields */
+  private static Document basicDocument() {
+    Document doc = new Document();
+    if (randomBoolean()) {
+      doc.add(new StoredField("field", "value"));
+    }
+    return doc;

Review comment:
       should this return an empty document directly so that we can more easily iterate on this test case when we stop writing stored fields when no stored fields are indexed?

##########
File path: lucene/core/src/java/org/apache/lucene/index/FieldInfos.java
##########
@@ -200,6 +204,11 @@ public boolean hasFreq() {
     return hasFreq;
   }
 
+  /** Returns true if any fields are indexed */
+  public boolean hasIndexed() {

Review comment:
       +1 to hasPostings

##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI
+
+      IndexWriterConfig writerConfig =
+          newIndexWriterConfig(new MockAnalyzer(random()))
+              .setCodec(new MinimalCodec(useCompoundFile, useDeletes))
+              .setUseCompoundFile(useCompoundFile);
+      if (!useCompoundFile) {
+        writerConfig.getMergePolicy().setNoCFSRatio(0.0);
+        writerConfig.getMergePolicy().setMaxCFSSegmentSizeMB(Double.POSITIVE_INFINITY);
+      }
+
+      int expectedNumDocs = 0;
+      try (IndexWriter writer = new IndexWriter(dir, writerConfig)) {
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.flush(); // create second segment
+        if (useDeletes && randomBoolean()) {

Review comment:
       can we rely less on randomization for coverage?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r778845058



##########
File path: lucene/core/src/java/org/apache/lucene/index/SegmentMerger.java
##########
@@ -203,8 +203,13 @@ private void mergeTerms(SegmentWriteState segmentWriteState, SegmentReadState se
         // Use the merge instance in order to reuse the same IndexInput for all terms
         normsMergeInstance = norms.getMergeInstance();
       }
-      try (FieldsConsumer consumer = codec.postingsFormat().fieldsConsumer(segmentWriteState)) {
-        consumer.merge(mergeState, normsMergeInstance);
+      try (FieldsConsumer consumer =
+          mergeState.mergeFieldInfos.hasIndexed()
+              ? codec.postingsFormat().fieldsConsumer(segmentWriteState)
+              : null) {
+        if (consumer != null) {
+          consumer.merge(mergeState, normsMergeInstance);
+        }

Review comment:
       fixed in 3d8a60d




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r778853159



##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI
+
+      IndexWriterConfig writerConfig =
+          newIndexWriterConfig(new MockAnalyzer(random()))
+              .setCodec(new MinimalCodec(useCompoundFile, useDeletes))
+              .setUseCompoundFile(useCompoundFile);
+      if (!useCompoundFile) {
+        writerConfig.getMergePolicy().setNoCFSRatio(0.0);
+        writerConfig.getMergePolicy().setMaxCFSSegmentSizeMB(Double.POSITIVE_INFINITY);
+      }
+
+      int expectedNumDocs = 0;
+      try (IndexWriter writer = new IndexWriter(dir, writerConfig)) {
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.flush(); // create second segment
+        if (useDeletes && randomBoolean()) {

Review comment:
       I've removed the `useDeletes` path. It's not necessary for the changes that I've introduced in this PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r778853488



##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI
+
+      IndexWriterConfig writerConfig =
+          newIndexWriterConfig(new MockAnalyzer(random()))
+              .setCodec(new MinimalCodec(useCompoundFile, useDeletes))
+              .setUseCompoundFile(useCompoundFile);
+      if (!useCompoundFile) {
+        writerConfig.getMergePolicy().setNoCFSRatio(0.0);
+        writerConfig.getMergePolicy().setMaxCFSSegmentSizeMB(Double.POSITIVE_INFINITY);
+      }
+
+      int expectedNumDocs = 0;
+      try (IndexWriter writer = new IndexWriter(dir, writerConfig)) {
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.flush(); // create second segment
+        if (useDeletes && randomBoolean()) {
+          writer.deleteDocuments(new MatchAllDocsQuery());
+          expectedNumDocs = 0;
+        }
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.forceMerge(1); // test merges
+        if (useDeletes && randomBoolean()) {
+          writer.deleteDocuments(new MatchAllDocsQuery());

Review comment:
       I've removed all this in d0a9f8c. It wasn't necessary to test my livedocs changes




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r778853694



##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI
+
+      IndexWriterConfig writerConfig =
+          newIndexWriterConfig(new MockAnalyzer(random()))
+              .setCodec(new MinimalCodec(useCompoundFile, useDeletes))
+              .setUseCompoundFile(useCompoundFile);
+      if (!useCompoundFile) {
+        writerConfig.getMergePolicy().setNoCFSRatio(0.0);
+        writerConfig.getMergePolicy().setMaxCFSSegmentSizeMB(Double.POSITIVE_INFINITY);
+      }
+
+      int expectedNumDocs = 0;
+      try (IndexWriter writer = new IndexWriter(dir, writerConfig)) {
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.flush(); // create second segment
+        if (useDeletes && randomBoolean()) {
+          writer.deleteDocuments(new MatchAllDocsQuery());
+          expectedNumDocs = 0;
+        }
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.forceMerge(1); // test merges
+        if (useDeletes && randomBoolean()) {
+          writer.deleteDocuments(new MatchAllDocsQuery());
+          expectedNumDocs = 0;
+        }
+        if (randomBoolean()) {
+          writer.commit();
+        }
+
+        try (DirectoryReader reader = DirectoryReader.open(writer)) {
+          assertEquals(expectedNumDocs, reader.numDocs());
+        }
+      }
+    }
+  }
+
+  /** returns a basic document with no indexed fields */
+  private static Document basicDocument() {
+    Document doc = new Document();
+    if (randomBoolean()) {
+      doc.add(new StoredField("field", "value"));
+    }
+    return doc;

Review comment:
       works for me (d0a9f8c)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r768409970



##########
File path: lucene/core/src/java/org/apache/lucene/index/SegmentCommitInfo.java
##########
@@ -244,7 +244,9 @@ public long sizeInBytes() throws IOException {
     // updates) and then maybe even be able to remove LiveDocsFormat.files().
 
     // Must separately add any live docs files:
-    info.getCodec().liveDocsFormat().files(this, files);
+    if (hasDeletions()) {
+      info.getCodec().liveDocsFormat().files(this, files);

Review comment:
       This now allows for a minimal codec that does not have a LiveDocsFormat when no deletes are being used. It was a tiny change to make, but perhaps out of scope for this PR?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on pull request #539:
URL: https://github.com/apache/lucene/pull/539#issuecomment-1005809668


   Thanks @jpountz!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r778843210



##########
File path: lucene/core/src/java/org/apache/lucene/index/FieldInfos.java
##########
@@ -200,6 +204,11 @@ public boolean hasFreq() {
     return hasFreq;
   }
 
+  /** Returns true if any fields are indexed */
+  public boolean hasIndexed() {

Review comment:
       fixed in 9f25571




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r778853977



##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI
+
+      IndexWriterConfig writerConfig =
+          newIndexWriterConfig(new MockAnalyzer(random()))
+              .setCodec(new MinimalCodec(useCompoundFile, useDeletes))
+              .setUseCompoundFile(useCompoundFile);
+      if (!useCompoundFile) {
+        writerConfig.getMergePolicy().setNoCFSRatio(0.0);
+        writerConfig.getMergePolicy().setMaxCFSSegmentSizeMB(Double.POSITIVE_INFINITY);
+      }
+
+      int expectedNumDocs = 0;
+      try (IndexWriter writer = new IndexWriter(dir, writerConfig)) {
+        writer.addDocument(basicDocument());
+        expectedNumDocs += 1;
+        writer.flush(); // create second segment

Review comment:
       ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r778852727



##########
File path: lucene/core/src/test/org/apache/lucene/codecs/TestMinimalCodec.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import static com.carrotsearch.randomizedtesting.RandomizedTest.randomBoolean;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.MockAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.BaseDirectoryWrapper;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+/**
+ * Tests to ensure that {@link Codec}s won't need to implement all formats in case where only a
+ * small subset of Lucene's functionality is used.
+ */
+public class TestMinimalCodec extends LuceneTestCase {
+
+  public void testMinimalCodec() throws IOException {
+    runMinimalCodecTest(false, false);
+    runMinimalCodecTest(false, true);
+    runMinimalCodecTest(true, true);
+    runMinimalCodecTest(true, false);
+  }
+
+  private void runMinimalCodecTest(boolean useCompoundFile, boolean useDeletes) throws IOException {
+    try (BaseDirectoryWrapper dir = newDirectory()) {
+      dir.setCheckIndexOnClose(false); // MinimalCodec is not registered with SPI

Review comment:
       done in d0a9f8c




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ywelsch commented on a change in pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
ywelsch commented on a change in pull request #539:
URL: https://github.com/apache/lucene/pull/539#discussion_r768407647



##########
File path: lucene/core/src/java/org/apache/lucene/index/FieldInfos.java
##########
@@ -200,6 +204,11 @@ public boolean hasFreq() {
     return hasFreq;
   }
 
+  /** Returns true if any fields are indexed */
+  public boolean hasIndexed() {

Review comment:
       I wasn't sure about the naming here. Perhaps `hasPostings` is a better name?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] jpountz merged pull request #539: LUCENE-10291: Only read/write postings when there is at least one indexed field

Posted by GitBox <gi...@apache.org>.
jpountz merged pull request #539:
URL: https://github.com/apache/lucene/pull/539


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org