You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Tim-Brooks (via GitHub)" <gi...@apache.org> on 2026/06/08 16:07:51 UTC

[PR] Allow multiple addBatch columns per field name [lucene]

Tim-Brooks opened a new pull request, #16219:
URL: https://github.com/apache/lucene/pull/16219

   Relaxes the one-column-per-field-name rule in addBatch so a field can be
   built from several columns contributing distinct indexing aspects — for
   example a stored BinaryColumn alongside an inverted BinaryColumn.
   
   processBatch now mirrors processDocument: schemas accumulate across
   same-named columns and FieldInfo is frozen once after the loop. Each
   aspect (inversion, stored, doc values, points, vectors) must still come
   from a single column, enforced via a per-field byte bitset of claimed
   aspects keyed off fieldGen.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "Tim-Brooks (via GitHub)" <gi...@apache.org>.
Tim-Brooks merged PR #16219:
URL: https://github.com/apache/lucene/pull/16219


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "mayya-sharipova (via GitHub)" <gi...@apache.org>.
mayya-sharipova commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3401634463


##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -740,14 +742,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and aspect set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnAspects = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing aspect must come from a single column for a given field name.
+      int columnAspects = ColumnValidation.aspectMask(fieldType);
+      int overlap = pf.columnAspects & columnAspects;
+      if (overlap != 0) {
         throw new IllegalArgumentException(
-            "ColumnBatch contains more than one column for field \""
+            "ColumnBatch has multiple columns for field \""
                 + fieldName
-                + "\"; multi-valued fields must use a single column with a sparse cursor");
+                + "\" claiming the same indexing aspect "
+                + ColumnValidation.aspectNames(overlap)
+                + "; each aspect must come from a single column.");

Review Comment:
   "must appear in **at most** one column"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "mayya-sharipova (via GitHub)" <gi...@apache.org>.
mayya-sharipova commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3401535620


##########
lucene/core/src/java/org/apache/lucene/document/column/ColumnValidation.java:
##########
@@ -32,11 +32,76 @@ public final class ColumnValidation {
 
   private ColumnValidation() {}
 
+  /** Inverted-index aspect ({@code indexOptions() != NONE}). */
+  static final int ASPECT_INVERSION = 1 << 0;
+
+  /** Stored-fields aspect ({@code stored()}). */
+  static final int ASPECT_STORED = 1 << 1;
+
+  /** Doc-values aspect ({@code docValuesType() != NONE}). */
+  static final int ASPECT_DOCVALUES = 1 << 2;
+
+  /** Points aspect ({@code pointDimensionCount() != 0}). */
+  static final int ASPECT_POINTS = 1 << 3;
+
+  /** KNN-vector aspect ({@code vectorDimension() != 0}). */
+  static final int ASPECT_VECTOR = 1 << 4;
+
+  /**
+   * Returns a bitmask of the indexing aspects ({@code ASPECT_*}) declared by {@code fieldType}.
+   * Used by the column-batch path to enforce that, when several columns share a field name, each
+   * aspect is carried by at most one column.
+   */
+  public static int aspectMask(IndexableFieldType fieldType) {
+    int mask = 0;
+    if (fieldType.indexOptions() != IndexOptions.NONE) {
+      mask |= ASPECT_INVERSION;
+    }
+    if (fieldType.stored()) {
+      mask |= ASPECT_STORED;
+    }
+    if (fieldType.docValuesType() != DocValuesType.NONE) {
+      mask |= ASPECT_DOCVALUES;
+    }
+    if (fieldType.pointDimensionCount() != 0) {
+      mask |= ASPECT_POINTS;
+    }
+    if (fieldType.vectorDimension() != 0) {
+      mask |= ASPECT_VECTOR;
+    }
+    return mask;
+  }
+
+  /** Returns a human-readable comma-separated list of the aspect names set in {@code mask}. */
+  public static String aspectNames(int mask) {
+    StringBuilder sb = new StringBuilder();
+    if ((mask & ASPECT_INVERSION) != 0) {
+      sb.append("inversion");
+    }
+    if ((mask & ASPECT_STORED) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("stored");
+    }
+    if ((mask & ASPECT_DOCVALUES) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("doc values");
+    }
+    if ((mask & ASPECT_POINTS) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("points");
+    }
+    if ((mask & ASPECT_VECTOR) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("vectors");
+    }
+    return "[" + sb + "]";
+  }
+
   /**
-   * Throws {@link IllegalArgumentException} if {@code fieldType} declares no indexing feature (no
+   * Throws {@link IllegalArgumentException} if {@code fieldType} declares no indexing aspect (no
    * doc values, no points, not stored, no index options, no vectors).
    */
-  public static void validateColumnHasIndexingFeature(
+  public static void validateColumnHasIndexingAspect(

Review Comment:
   At first I had difficult to understand what is meant by "aspect", I liked the previous name "feature" better



##########
lucene/core/src/java/org/apache/lucene/document/column/ColumnValidation.java:
##########
@@ -32,11 +32,76 @@ public final class ColumnValidation {
 
   private ColumnValidation() {}
 
+  /** Inverted-index aspect ({@code indexOptions() != NONE}). */
+  static final int ASPECT_INVERSION = 1 << 0;
+
+  /** Stored-fields aspect ({@code stored()}). */
+  static final int ASPECT_STORED = 1 << 1;
+
+  /** Doc-values aspect ({@code docValuesType() != NONE}). */
+  static final int ASPECT_DOCVALUES = 1 << 2;
+
+  /** Points aspect ({@code pointDimensionCount() != 0}). */
+  static final int ASPECT_POINTS = 1 << 3;
+
+  /** KNN-vector aspect ({@code vectorDimension() != 0}). */
+  static final int ASPECT_VECTOR = 1 << 4;
+
+  /**
+   * Returns a bitmask of the indexing aspects ({@code ASPECT_*}) declared by {@code fieldType}.
+   * Used by the column-batch path to enforce that, when several columns share a field name, each
+   * aspect is carried by at most one column.
+   */
+  public static int aspectMask(IndexableFieldType fieldType) {
+    int mask = 0;
+    if (fieldType.indexOptions() != IndexOptions.NONE) {
+      mask |= ASPECT_INVERSION;
+    }
+    if (fieldType.stored()) {
+      mask |= ASPECT_STORED;
+    }
+    if (fieldType.docValuesType() != DocValuesType.NONE) {
+      mask |= ASPECT_DOCVALUES;
+    }
+    if (fieldType.pointDimensionCount() != 0) {
+      mask |= ASPECT_POINTS;
+    }
+    if (fieldType.vectorDimension() != 0) {
+      mask |= ASPECT_VECTOR;
+    }
+    return mask;
+  }
+
+  /** Returns a human-readable comma-separated list of the aspect names set in {@code mask}. */
+  public static String aspectNames(int mask) {
+    StringBuilder sb = new StringBuilder();
+    if ((mask & ASPECT_INVERSION) != 0) {
+      sb.append("inversion");
+    }
+    if ((mask & ASPECT_STORED) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("stored");
+    }
+    if ((mask & ASPECT_DOCVALUES) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("doc values");
+    }
+    if ((mask & ASPECT_POINTS) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("points");
+    }
+    if ((mask & ASPECT_VECTOR) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("vectors");
+    }
+    return "[" + sb + "]";
+  }
+
   /**
-   * Throws {@link IllegalArgumentException} if {@code fieldType} declares no indexing feature (no
+   * Throws {@link IllegalArgumentException} if {@code fieldType} declares no indexing aspect (no
    * doc values, no points, not stored, no index options, no vectors).
    */
-  public static void validateColumnHasIndexingFeature(
+  public static void validateColumnHasIndexingAspect(

Review Comment:
   At first I had difficulty to understand what is meant by "aspect", I liked the previous name "feature" better



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "mayya-sharipova (via GitHub)" <gi...@apache.org>.
mayya-sharipova commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3405971053


##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -740,14 +742,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and aspect set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnAspects = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing aspect must come from a single column for a given field name.
+      int columnAspects = ColumnValidation.aspectMask(fieldType);
+      int overlap = pf.columnAspects & columnAspects;

Review Comment:
   Should we only calculate overlap if pf.fieldGen == batchGen?
   
   ```java
   if (pf.fieldGen != batchGen) {
         pf.fieldGen = batchGen;
         pf.columnAspects = 0;
         pf.schema.reset(baseDocID);
         fields[uniqueFieldCount++] = pf;
     } else {
         int overlap = pf.columnAspects & ColumnValidation.aspectMask(fieldType);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "mayya-sharipova (via GitHub)" <gi...@apache.org>.
mayya-sharipova commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3401634463


##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -740,14 +742,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and aspect set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnAspects = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing aspect must come from a single column for a given field name.
+      int columnAspects = ColumnValidation.aspectMask(fieldType);
+      int overlap = pf.columnAspects & columnAspects;
+      if (overlap != 0) {
         throw new IllegalArgumentException(
-            "ColumnBatch contains more than one column for field \""
+            "ColumnBatch has multiple columns for field \""
                 + fieldName
-                + "\"; multi-valued fields must use a single column with a sparse cursor");
+                + "\" claiming the same indexing aspect "
+                + ColumnValidation.aspectNames(overlap)
+                + "; each aspect must come from a single column.");

Review Comment:
   nit: "must come  from **at most** one column"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "Tim-Brooks (via GitHub)" <gi...@apache.org>.
Tim-Brooks commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3402908658


##########
lucene/core/src/java/org/apache/lucene/document/column/ColumnValidation.java:
##########
@@ -32,11 +32,76 @@ public final class ColumnValidation {
 
   private ColumnValidation() {}
 
+  /** Inverted-index aspect ({@code indexOptions() != NONE}). */
+  static final int ASPECT_INVERSION = 1 << 0;
+
+  /** Stored-fields aspect ({@code stored()}). */
+  static final int ASPECT_STORED = 1 << 1;
+
+  /** Doc-values aspect ({@code docValuesType() != NONE}). */
+  static final int ASPECT_DOCVALUES = 1 << 2;
+
+  /** Points aspect ({@code pointDimensionCount() != 0}). */
+  static final int ASPECT_POINTS = 1 << 3;
+
+  /** KNN-vector aspect ({@code vectorDimension() != 0}). */
+  static final int ASPECT_VECTOR = 1 << 4;
+
+  /**
+   * Returns a bitmask of the indexing aspects ({@code ASPECT_*}) declared by {@code fieldType}.
+   * Used by the column-batch path to enforce that, when several columns share a field name, each
+   * aspect is carried by at most one column.
+   */
+  public static int aspectMask(IndexableFieldType fieldType) {
+    int mask = 0;
+    if (fieldType.indexOptions() != IndexOptions.NONE) {
+      mask |= ASPECT_INVERSION;
+    }
+    if (fieldType.stored()) {
+      mask |= ASPECT_STORED;
+    }
+    if (fieldType.docValuesType() != DocValuesType.NONE) {
+      mask |= ASPECT_DOCVALUES;
+    }
+    if (fieldType.pointDimensionCount() != 0) {
+      mask |= ASPECT_POINTS;
+    }
+    if (fieldType.vectorDimension() != 0) {
+      mask |= ASPECT_VECTOR;
+    }
+    return mask;
+  }
+
+  /** Returns a human-readable comma-separated list of the aspect names set in {@code mask}. */
+  public static String aspectNames(int mask) {
+    StringBuilder sb = new StringBuilder();
+    if ((mask & ASPECT_INVERSION) != 0) {
+      sb.append("inversion");
+    }
+    if ((mask & ASPECT_STORED) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("stored");
+    }
+    if ((mask & ASPECT_DOCVALUES) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("doc values");
+    }
+    if ((mask & ASPECT_POINTS) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("points");
+    }
+    if ((mask & ASPECT_VECTOR) != 0) {
+      if (!sb.isEmpty()) sb.append(", ");
+      sb.append("vectors");
+    }
+    return "[" + sb + "]";
+  }
+
   /**
-   * Throws {@link IllegalArgumentException} if {@code fieldType} declares no indexing feature (no
+   * Throws {@link IllegalArgumentException} if {@code fieldType} declares no indexing aspect (no
    * doc values, no points, not stored, no index options, no vectors).
    */
-  public static void validateColumnHasIndexingFeature(
+  public static void validateColumnHasIndexingAspect(

Review Comment:
   I was split. So I'm fine with "feature" and changed it back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "mayya-sharipova (via GitHub)" <gi...@apache.org>.
mayya-sharipova commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3401634463


##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -740,14 +742,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and aspect set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnAspects = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing aspect must come from a single column for a given field name.
+      int columnAspects = ColumnValidation.aspectMask(fieldType);
+      int overlap = pf.columnAspects & columnAspects;
+      if (overlap != 0) {
         throw new IllegalArgumentException(
-            "ColumnBatch contains more than one column for field \""
+            "ColumnBatch has multiple columns for field \""
                 + fieldName
-                + "\"; multi-valued fields must use a single column with a sparse cursor");
+                + "\" claiming the same indexing aspect "
+                + ColumnValidation.aspectNames(overlap)
+                + "; each aspect must come from a single column.");

Review Comment:
   "must come  from **at most** one column"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "mayya-sharipova (via GitHub)" <gi...@apache.org>.
mayya-sharipova commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3405971053


##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -740,14 +742,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and aspect set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnAspects = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing aspect must come from a single column for a given field name.
+      int columnAspects = ColumnValidation.aspectMask(fieldType);
+      int overlap = pf.columnAspects & columnAspects;

Review Comment:
   Should we only calculate overlap if pf.fieldGen == batchGen?
   
   ```java
   if (pf.fieldGen != batchGen) {
         pf.fieldGen = batchGen;
         pf.columnAspects = 0;
         pf.schema.reset(baseDocID);
         fields[uniqueFieldCount++] = pf;
     } else {
         int overlap = pf.columnAspects & ColumnValidation.aspectMask(fieldType);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "mayya-sharipova (via GitHub)" <gi...@apache.org>.
mayya-sharipova commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3405979215


##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -744,14 +746,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and feature set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnFeatures = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing feature must come from a single column for a given field name.
+      int columnFeatures = ColumnValidation.featureMask(fieldType);
+      int overlap = pf.columnFeatures & columnFeatures;

Review Comment:
   Should we only calculate overlap if pf.fieldGen == batchGen?
   
   if (pf.fieldGen != batchGen) {
         pf.fieldGen = batchGen;
         pf.columnAspects = 0;
         pf.schema.reset(baseDocID);
         fields[uniqueFieldCount++] = pf;
     } else {
         int overlap = pf.columnAspects & ColumnValidation.aspectMask(fieldType);



##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -744,14 +746,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and feature set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnFeatures = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing feature must come from a single column for a given field name.
+      int columnFeatures = ColumnValidation.featureMask(fieldType);
+      int overlap = pf.columnFeatures & columnFeatures;

Review Comment:
   Should we only calculate overlap if pf.fieldGen == batchGen?
   
   ```
   if (pf.fieldGen != batchGen) {
         pf.fieldGen = batchGen;
         pf.columnAspects = 0;
         pf.schema.reset(baseDocID);
         fields[uniqueFieldCount++] = pf;
     } else {
         int overlap = pf.columnAspects & ColumnValidation.aspectMask(fieldType);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "mayya-sharipova (via GitHub)" <gi...@apache.org>.
mayya-sharipova commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3405979215


##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -744,14 +746,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and feature set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnFeatures = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing feature must come from a single column for a given field name.
+      int columnFeatures = ColumnValidation.featureMask(fieldType);
+      int overlap = pf.columnFeatures & columnFeatures;

Review Comment:
   Should we only calculate overlap if pf.fieldGen == batchGen?
   
   ```
   if (pf.fieldGen != batchGen) {
         pf.fieldGen = batchGen;
         pf.columnAspects = 0;
         pf.schema.reset(baseDocID);
         fields[uniqueFieldCount++] = pf;
     } else {
        ...
         int overlap = pf.columnAspects & ColumnValidation.aspectMask(fieldType);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Allow multiple addBatch columns per field name [lucene]

Posted by "Tim-Brooks (via GitHub)" <gi...@apache.org>.
Tim-Brooks commented on code in PR #16219:
URL: https://github.com/apache/lucene/pull/16219#discussion_r3402924651


##########
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:
##########
@@ -740,14 +742,34 @@ void processBatch(int baseDocID, ColumnBatch columnBatch) throws IOException {
       }
       docFields[columnIdx++] = pf;
 
-      if (pf.fieldGen == batchGen) {
+      if (pf.fieldGen != batchGen) {
+        // First column for this field name in this batch: start a fresh schema and aspect set, and
+        // collect the field once so its FieldInfo is initialized/validated after the loop.
+        pf.fieldGen = batchGen;
+        pf.columnAspects = 0;
+        pf.schema.reset(baseDocID);
+        fields[uniqueFieldCount++] = pf;
+      }
+
+      // Each indexing aspect must come from a single column for a given field name.
+      int columnAspects = ColumnValidation.aspectMask(fieldType);
+      int overlap = pf.columnAspects & columnAspects;
+      if (overlap != 0) {
         throw new IllegalArgumentException(
-            "ColumnBatch contains more than one column for field \""
+            "ColumnBatch has multiple columns for field \""
                 + fieldName
-                + "\"; multi-valued fields must use a single column with a sparse cursor");
+                + "\" claiming the same indexing aspect "
+                + ColumnValidation.aspectNames(overlap)
+                + "; each aspect must come from a single column.");

Review Comment:
   I went similar but slightly different: "each feature may appear in at most one column". Hope that works.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org