You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gargvishesh (via GitHub)" <gi...@apache.org> on 2024/10/15 05:07:37 UTC

[PR] Add multi-valued check for partition dimensions (druid)

gargvishesh opened a new pull request, #17344:
URL: https://github.com/apache/druid/pull/17344

   MSQ currently supports only single-valued string dimensions as partition keys. This patch adds a check to ensure that partition keys are single-valued in case this info is available by virtue of segment download for schema inference. For the case where segments aren't downloaded, since this info is only available at runtime, this patch makes no changes; the partitioning eventually ends up getting switched to dynamic type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1804707020


##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,66 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * Returns true even when granularitySpec is not known since rollup is then decided based on segment analysis.
+   */
+  static boolean isPossiblyRollup(@Nullable ClientCompactionTaskGranularitySpec granularitySpec)
+  {
+    return !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+  }
+
+  static boolean isRangePartitioned(@Nullable TuningConfig tuningConfig)
+  {
+    return tuningConfig != null && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+  }
+
+  static boolean isRollupOnStringDimension(
+      DimensionsSpec dimensionsSpec,
+      @Nullable ClientCompactionTaskGranularitySpec granularitySpec
+  )
+  {
+    if (!isPossiblyRollup(granularitySpec)) {
+      return false;
+    }
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(dim -> ColumnType.STRING.equals(dim.getColumnType()));
+  }
+
+  static boolean isPartitionedOnStringDimension(DimensionsSpec dimensionsSpec, @Nullable TuningConfig tuningConfig)
+  {
+    if (!isRangePartitioned(tuningConfig)) {
+      return false;
+    }
+    DimensionRangePartitionsSpec partitionsSpec = (DimensionRangePartitionsSpec) tuningConfig.getPartitionsSpec();
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(
+                             dim -> ColumnType.STRING.equals(dim.getColumnType())
+                                    && partitionsSpec.getPartitionDimensions().contains(dim.getName())
+                         );
+  }
+
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  public static boolean needMultiValuedDimensions(CompactionTask compactionTask)

Review Comment:
   public is needed since the tests reside in the MSQ extension



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on PR #17344:
URL: https://github.com/apache/druid/pull/17344#issuecomment-2413166083

   Right. So this case can happen when there is no need to download the segments, ie. user specifies dimensions, metrics, granularity and rollup in the compaction spec itself, but one of the dimensions happens to be MVD.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1805974309


##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,45 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  boolean needMultiValuedDimensions()

Review Comment:
   Updated to `identifyMultiValuedDimensions`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1800470253


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java:
##########
@@ -130,7 +130,7 @@ public MSQCompactionRunner(
    * The following configs aren't supported:
    * <ul>
    * <li>partitionsSpec of type HashedParititionsSpec.</li>
-   * <li>'range' partitionsSpec with non-string partition dimensions.</li>
+   * <li>'range' partitionsSpec with non-single-valued-string partition dimensions.</li>

Review Comment:
   ```suggestion
      * <li>'range' partitionsSpec with multi-valued or non-string partition dimensions.</li>
   ```



##########
server/src/main/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfo.java:
##########
@@ -138,13 +141,19 @@ private static CompactionConfigValidationResult compactionConfigSupportedByMSQEn
 
   /**
    * Validate that partitionSpec is either 'dynamic` or 'range'. If 'dynamic', ensure 'maxTotalRows' is null. If range
-   * ensure all partition columns are of string type.
+   * ensure all partition columns are of single-valued string type .

Review Comment:
   ```suggestion
      * ensure all partition columns are single-valued and have type string.
   ```



##########
server/src/main/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfo.java:
##########
@@ -138,13 +141,19 @@ private static CompactionConfigValidationResult compactionConfigSupportedByMSQEn
 
   /**
    * Validate that partitionSpec is either 'dynamic` or 'range'. If 'dynamic', ensure 'maxTotalRows' is null. If range
-   * ensure all partition columns are of string type.
+   * ensure all partition columns are of single-valued string type .
    */
   public static CompactionConfigValidationResult validatePartitionsSpecForMSQ(
       @Nullable PartitionsSpec partitionsSpec,
-      @Nullable List<DimensionSchema> dimensionSchemas
+      @Nullable List<DimensionSchema> dimensionSchemas,
+      @Nullable Set<String> multiValuedDimensions
   )
   {
+    if (partitionsSpec == null) {
+      return CompactionConfigValidationResult.failure(
+          "MSQ: tuningConfig.partitionsSpec must be set"

Review Comment:
   Why do we need this? I don't think specifying `partitionsSpec` is a requirement in native engine?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java:
##########
@@ -153,11 +153,15 @@ public CompactionConfigValidationResult validateCompactionTask(
       );
     }
     List<CompactionConfigValidationResult> validationResults = new ArrayList<>();
+    DataSchema dataSchema = Iterables.getOnlyElement(intervalToDataSchemaMap.values());

Review Comment:
   Could you please add a comment on why this is always single valued? I think we had missed adding the comment in the previous PR.



##########
server/src/main/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfo.java:
##########
@@ -163,17 +172,26 @@ public static CompactionConfigValidationResult validatePartitionsSpecForMSQ(
       Map<String, DimensionSchema> dimensionSchemaMap = dimensionSchemas.stream().collect(
           Collectors.toMap(DimensionSchema::getName, Function.identity())
       );
-      Optional<String> nonStringDimension = ((DimensionRangePartitionsSpec) partitionsSpec)
+      Optional<String> unsupportedDimension = ((DimensionRangePartitionsSpec) partitionsSpec)
           .getPartitionDimensions()
           .stream()
-          .filter(dim -> !ColumnType.STRING.equals(dimensionSchemaMap.get(dim).getColumnType()))
+          .filter(dim -> !ColumnType.STRING.equals(dimensionSchemaMap.get(dim).getColumnType())
+                         || multiValuedDimensions != null && multiValuedDimensions.contains(dim))

Review Comment:
   For clearer code, let's leave the whole check for non string dimension as-is.
   Add a separate loop for checking multi-valued dimensions.



##########
server/src/main/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfo.java:
##########
@@ -107,7 +108,7 @@ public static CompactionConfigValidationResult validateCompactionConfig(
    * Checks if the provided compaction config is supported by MSQ. The following configs aren't supported:
    * <ul>
    * <li>partitionsSpec of type HashedParititionsSpec.</li>
-   * <li>'range' partitionsSpec with non-string partition dimensions.</li>
+   * <li>'range' partitionsSpec with non-single-valued-string partition dimensions.</li>

Review Comment:
   ```suggestion
      * <li>'range' partitionsSpec with multi-valued or non-string partition dimensions.</li>
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1805876609


##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,45 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  boolean needMultiValuedDimensions()
+  {
+    if (compactionRunner instanceof NativeCompactionRunner) {
+      return false;
+    }
+    // Rollup can be true even when granularitySpec is not known since rollup is then decided based on segment analysis
+    boolean isPossiblyRollup = !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+    boolean isRangePartitioned = tuningConfig != null
+                                 && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+
+    if (dimensionsSpec == null || dimensionsSpec.getDimensions().isEmpty()) {
+      return (isPossiblyRollup || isRangePartitioned);

Review Comment:
   ```suggestion
         return isPossiblyRollup || isRangePartitioned;
   ```



##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,45 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.

Review Comment:
   ```suggestion
      * Checks if multi-valued string dimensions need to be analyzed by downloading the segments.
      * This method returns true only for MSQ engine when either of the following holds true:
      * <ul>
      * <li> Range partitioning is done on a string dimension or an unknown dimension
      * (since MSQ does not support partitioning on a multi-valued string dimension) </li>
      * <li> Rollup is done on a string dimension or an unknown dimension
      * (since MSQ requires multi-valued string dimensions to be converted to arrays for rollup) </li>
      * </ul>
      * @return false for native engine, true for MSQ engine only when partitioning or rollup is done on a string or unknown dimension.
   ```



##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,45 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  boolean needMultiValuedDimensions()
+  {
+    if (compactionRunner instanceof NativeCompactionRunner) {
+      return false;
+    }
+    // Rollup can be true even when granularitySpec is not known since rollup is then decided based on segment analysis
+    boolean isPossiblyRollup = !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+    boolean isRangePartitioned = tuningConfig != null
+                                 && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+
+    if (dimensionsSpec == null || dimensionsSpec.getDimensions().isEmpty()) {
+      return (isPossiblyRollup || isRangePartitioned);
+    } else {
+      boolean isRollupOnStringDimension = isPossiblyRollup &&
+                                          dimensionsSpec.getDimensions()
+                                                        .stream()
+                                                        .anyMatch(dim -> ColumnType.STRING.equals(
+                                                            dim.getColumnType()));

Review Comment:
   formatting:
   ```suggestion
                                                           .anyMatch(dim -> ColumnType.STRING.equals(dim.getColumnType()));
   ```



##########
indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java:
##########
@@ -2015,6 +2100,37 @@ public SettableColumnValueSelector<?> makeNewSettableColumnValueSelector()
 
   }
 
+  /**
+   * A class to mimic MSQCompactionRunner behaviour, since the actual class resides in the MSQ extn

Review Comment:
   How are we mimicking the behaviour? The methods seem to return only null.



##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,45 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  boolean needMultiValuedDimensions()

Review Comment:
   Can this be renamed to something like `needToIdentifyMultiValuedDimensions` or `shouldAnalyzeMultiValuedDimensions` (to correspond to segment analyzer)?
   Just `needMvd` seems ambiguous.



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java:
##########
@@ -175,6 +187,31 @@ public CompactionConfigValidationResult validateCompactionTask(
                             .orElse(CompactionConfigValidationResult.success());
   }
 
+  private CompactionConfigValidationResult validatePartitionDimensionsNotMultivalued(
+      PartitionsSpec partitionsSpec,
+      DimensionsSpec dimensionsSpec,
+      Set<String> multiValuedDimensions
+  )
+  {
+    List<String> dimensionSchemas = dimensionsSpec.getDimensionNames();
+    if (partitionsSpec instanceof DimensionRangePartitionsSpec
+        && dimensionSchemas != null
+        && multiValuedDimensions != null) {

Review Comment:
   Check for `multiValuedDimensions` empty too?



##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,45 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  boolean needMultiValuedDimensions()
+  {
+    if (compactionRunner instanceof NativeCompactionRunner) {
+      return false;
+    }
+    // Rollup can be true even when granularitySpec is not known since rollup is then decided based on segment analysis
+    boolean isPossiblyRollup = !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+    boolean isRangePartitioned = tuningConfig != null
+                                 && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+
+    if (dimensionsSpec == null || dimensionsSpec.getDimensions().isEmpty()) {
+      return (isPossiblyRollup || isRangePartitioned);
+    } else {
+      boolean isRollupOnStringDimension = isPossiblyRollup &&
+                                          dimensionsSpec.getDimensions()
+                                                        .stream()
+                                                        .anyMatch(dim -> ColumnType.STRING.equals(
+                                                            dim.getColumnType()));
+
+      boolean isPartitionedOnStringDimension =
+          isRangePartitioned &&
+          dimensionsSpec.getDimensions()
+                        .stream()
+                        .anyMatch(
+                            dim -> ColumnType.STRING.equals(
+                                dim.getColumnType())

Review Comment:
   formatting:
   ```suggestion
                               dim -> ColumnType.STRING.equals(dim.getColumnType())
   ```



##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,45 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  boolean needMultiValuedDimensions()
+  {
+    if (compactionRunner instanceof NativeCompactionRunner) {
+      return false;
+    }
+    // Rollup can be true even when granularitySpec is not known since rollup is then decided based on segment analysis
+    boolean isPossiblyRollup = !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));

Review Comment:
   Trying to remove the double negative:
   ```suggestion
       final boolean isPossiblyRollup = granularitySpec == null || !Boolean.FALSE.equals(granularitySpec.isRollup());
   ```



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java:
##########
@@ -175,6 +187,31 @@ public CompactionConfigValidationResult validateCompactionTask(
                             .orElse(CompactionConfigValidationResult.success());
   }
 
+  private CompactionConfigValidationResult validatePartitionDimensionsNotMultivalued(

Review Comment:
   ```suggestion
     private CompactionConfigValidationResult validatePartitionDimensionsAreNotMultiValued(
   ```



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java:
##########
@@ -153,13 +155,23 @@ public CompactionConfigValidationResult validateCompactionTask(
       );
     }
     List<CompactionConfigValidationResult> validationResults = new ArrayList<>();
+    DataSchema dataSchema = Iterables.getOnlyElement(intervalToDataSchemaMap.values());
     if (compactionTask.getTuningConfig() != null) {
       validationResults.add(
           ClientCompactionRunnerInfo.validatePartitionsSpecForMSQ(
               compactionTask.getTuningConfig().getPartitionsSpec(),
-              Iterables.getOnlyElement(intervalToDataSchemaMap.values()).getDimensionsSpec().getDimensions()
+              dataSchema.getDimensionsSpec().getDimensions()
           )
       );
+      validationResults.add(
+          validatePartitionDimensionsNotMultivalued(
+              compactionTask.getTuningConfig().getPartitionsSpec(),
+              dataSchema.getDimensionsSpec(),
+              dataSchema instanceof CombinedDataSchema
+              ? ((CombinedDataSchema) dataSchema).getMultiValuedDimensions()
+              : null
+          ));

Review Comment:
   format:
   ```suggestion
             )
         );
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1802600206


##########
server/src/main/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfo.java:
##########
@@ -163,17 +172,26 @@ public static CompactionConfigValidationResult validatePartitionsSpecForMSQ(
       Map<String, DimensionSchema> dimensionSchemaMap = dimensionSchemas.stream().collect(
           Collectors.toMap(DimensionSchema::getName, Function.identity())
       );
-      Optional<String> nonStringDimension = ((DimensionRangePartitionsSpec) partitionsSpec)
+      Optional<String> unsupportedDimension = ((DimensionRangePartitionsSpec) partitionsSpec)
           .getPartitionDimensions()
           .stream()
-          .filter(dim -> !ColumnType.STRING.equals(dimensionSchemaMap.get(dim).getColumnType()))
+          .filter(dim -> !ColumnType.STRING.equals(dimensionSchemaMap.get(dim).getColumnType())
+                         || multiValuedDimensions != null && multiValuedDimensions.contains(dim))

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz merged PR #17344:
URL: https://github.com/apache/druid/pull/17344


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #17344:
URL: https://github.com/apache/druid/pull/17344#issuecomment-2413159631

   > This in fact also means that compaction will keeping continuing in a loop. I was thinking about looking for a solution to this problem post this PR.
   
   Oh, that can be a problem. When can this happen?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1802609524


##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -796,8 +858,11 @@ static class ExistingSegmentAnalyzer
 
     private boolean shouldFetchSegments()
     {
-      // Don't fetch segments for just needMultiValueDimensions
-      return needRollup || needQueryGranularity || needDimensionsSpec || needMetricsSpec;
+      // Segments are downloaded even when just needMultiValuedDimensions=true since MSQ switches to dynamic
+      // partitioning on finding any 'range' partition dimension to be multi-valued at runtime, which ends up in a
+      // mismatch between the compaction config and the actual segments (lastCompactionState), leading to
+      // repeated compactions.

Review Comment:
   Please put this in the javadoc of this method instead.



##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,66 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * Returns true even when granularitySpec is not known since rollup is then decided based on segment analysis.
+   */
+  static boolean isPossiblyRollup(@Nullable ClientCompactionTaskGranularitySpec granularitySpec)
+  {
+    return !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+  }
+
+  static boolean isRangePartitioned(@Nullable TuningConfig tuningConfig)
+  {
+    return tuningConfig != null && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+  }
+
+  static boolean isRollupOnStringDimension(
+      DimensionsSpec dimensionsSpec,
+      @Nullable ClientCompactionTaskGranularitySpec granularitySpec
+  )
+  {
+    if (!isPossiblyRollup(granularitySpec)) {
+      return false;
+    }
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(dim -> ColumnType.STRING.equals(dim.getColumnType()));
+  }
+
+  static boolean isPartitionedOnStringDimension(DimensionsSpec dimensionsSpec, @Nullable TuningConfig tuningConfig)

Review Comment:
   I don't think all of these need to be separate methods as they are not being reused anywhere afaict. They can just be variables inside the `needMVD` method.



##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,66 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * Returns true even when granularitySpec is not known since rollup is then decided based on segment analysis.
+   */
+  static boolean isPossiblyRollup(@Nullable ClientCompactionTaskGranularitySpec granularitySpec)
+  {
+    return !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+  }
+
+  static boolean isRangePartitioned(@Nullable TuningConfig tuningConfig)
+  {
+    return tuningConfig != null && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+  }
+
+  static boolean isRollupOnStringDimension(
+      DimensionsSpec dimensionsSpec,
+      @Nullable ClientCompactionTaskGranularitySpec granularitySpec
+  )
+  {
+    if (!isPossiblyRollup(granularitySpec)) {
+      return false;
+    }
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(dim -> ColumnType.STRING.equals(dim.getColumnType()));
+  }
+
+  static boolean isPartitionedOnStringDimension(DimensionsSpec dimensionsSpec, @Nullable TuningConfig tuningConfig)
+  {
+    if (!isRangePartitioned(tuningConfig)) {
+      return false;
+    }
+    DimensionRangePartitionsSpec partitionsSpec = (DimensionRangePartitionsSpec) tuningConfig.getPartitionsSpec();
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(
+                             dim -> ColumnType.STRING.equals(dim.getColumnType())
+                                    && partitionsSpec.getPartitionDimensions().contains(dim.getName())
+                         );
+  }
+
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  public static boolean needMultiValuedDimensions(CompactionTask compactionTask)

Review Comment:
   No point making this method static if it requires passing the `CompactionTask` into it.
   Also, instead of making this method public or package private, `CompactionTask` should instead expose a method `needsToDownloadSegments()` or something. That can internally call this one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1804578970


##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,66 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * Returns true even when granularitySpec is not known since rollup is then decided based on segment analysis.
+   */
+  static boolean isPossiblyRollup(@Nullable ClientCompactionTaskGranularitySpec granularitySpec)
+  {
+    return !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+  }
+
+  static boolean isRangePartitioned(@Nullable TuningConfig tuningConfig)
+  {
+    return tuningConfig != null && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+  }
+
+  static boolean isRollupOnStringDimension(
+      DimensionsSpec dimensionsSpec,
+      @Nullable ClientCompactionTaskGranularitySpec granularitySpec
+  )
+  {
+    if (!isPossiblyRollup(granularitySpec)) {
+      return false;
+    }
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(dim -> ColumnType.STRING.equals(dim.getColumnType()));
+  }
+
+  static boolean isPartitionedOnStringDimension(DimensionsSpec dimensionsSpec, @Nullable TuningConfig tuningConfig)
+  {
+    if (!isRangePartitioned(tuningConfig)) {
+      return false;
+    }
+    DimensionRangePartitionsSpec partitionsSpec = (DimensionRangePartitionsSpec) tuningConfig.getPartitionsSpec();
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(
+                             dim -> ColumnType.STRING.equals(dim.getColumnType())
+                                    && partitionsSpec.getPartitionDimensions().contains(dim.getName())
+                         );
+  }
+
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  public static boolean needMultiValuedDimensions(CompactionTask compactionTask)

Review Comment:
   Just relooked at code. Only segmentGranularity is computed later, so all need* flags and an additional new needsDownload flag can be computed in the CompactionTask constructor itself. The needsDownload can have a public getter for tests. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on PR #17344:
URL: https://github.com/apache/druid/pull/17344#issuecomment-2413138665

   @kfaraz: yes, if MVD fields are found, there is just a log that states shard spec is changed to `NumberedShardSpec` type: https://github.com/apache/druid/blob/c2149d59a7da1ac4cbb02aaa63a52cb326306dc5/extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java#L1104
   
   This in fact also means that compaction will keeping continuing in a loop. I was thinking about looking for a solution to this problem post this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #17344:
URL: https://github.com/apache/druid/pull/17344#issuecomment-2413173864

   Can we ensure we download the segments in this case too?
   What would be the drawback?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on PR #17344:
URL: https://github.com/apache/druid/pull/17344#issuecomment-2413199658

   Drawback was mainly the high cost of downloading segments. But yes I think this would be the only viable way for now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #17344:
URL: https://github.com/apache/druid/pull/17344#issuecomment-2412918170

   > the partitioning eventually ends up getting switched to dynamic type.
   
   @gargvishesh , is this something MSQ always does silently?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1800642797


##########
server/src/main/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfo.java:
##########
@@ -138,13 +141,19 @@ private static CompactionConfigValidationResult compactionConfigSupportedByMSQEn
 
   /**
    * Validate that partitionSpec is either 'dynamic` or 'range'. If 'dynamic', ensure 'maxTotalRows' is null. If range
-   * ensure all partition columns are of string type.
+   * ensure all partition columns are of single-valued string type .
    */
   public static CompactionConfigValidationResult validatePartitionsSpecForMSQ(
       @Nullable PartitionsSpec partitionsSpec,
-      @Nullable List<DimensionSchema> dimensionSchemas
+      @Nullable List<DimensionSchema> dimensionSchemas,
+      @Nullable Set<String> multiValuedDimensions
   )
   {
+    if (partitionsSpec == null) {
+      return CompactionConfigValidationResult.failure(
+          "MSQ: tuningConfig.partitionsSpec must be set"

Review Comment:
   In native, a default spec is applied by the index task based on certain field values: https://github.com/apache/druid/blob/c2149d59a7da1ac4cbb02aaa63a52cb326306dc5/indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java#L1247
   
   This is not the case with MSQ and the task fails. 
   
   In the web-ui for compaction, partitionsSpec is populated with some value by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1805972605


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java:
##########
@@ -175,6 +187,31 @@ public CompactionConfigValidationResult validateCompactionTask(
                             .orElse(CompactionConfigValidationResult.success());
   }
 
+  private CompactionConfigValidationResult validatePartitionDimensionsNotMultivalued(
+      PartitionsSpec partitionsSpec,
+      DimensionsSpec dimensionsSpec,
+      Set<String> multiValuedDimensions
+  )
+  {
+    List<String> dimensionSchemas = dimensionsSpec.getDimensionNames();
+    if (partitionsSpec instanceof DimensionRangePartitionsSpec
+        && dimensionSchemas != null
+        && multiValuedDimensions != null) {

Review Comment:
   Yes makes it a bit more efficient. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1805973578


##########
indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java:
##########
@@ -2015,6 +2100,37 @@ public SettableColumnValueSelector<?> makeNewSettableColumnValueSelector()
 
   }
 
+  /**
+   * A class to mimic MSQCompactionRunner behaviour, since the actual class resides in the MSQ extn

Review Comment:
   Just mimics for validations. Updated the comment 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1804702632


##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,66 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * Returns true even when granularitySpec is not known since rollup is then decided based on segment analysis.
+   */
+  static boolean isPossiblyRollup(@Nullable ClientCompactionTaskGranularitySpec granularitySpec)
+  {
+    return !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+  }
+
+  static boolean isRangePartitioned(@Nullable TuningConfig tuningConfig)
+  {
+    return tuningConfig != null && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+  }
+
+  static boolean isRollupOnStringDimension(
+      DimensionsSpec dimensionsSpec,
+      @Nullable ClientCompactionTaskGranularitySpec granularitySpec
+  )
+  {
+    if (!isPossiblyRollup(granularitySpec)) {
+      return false;
+    }
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(dim -> ColumnType.STRING.equals(dim.getColumnType()));
+  }
+
+  static boolean isPartitionedOnStringDimension(DimensionsSpec dimensionsSpec, @Nullable TuningConfig tuningConfig)
+  {
+    if (!isRangePartitioned(tuningConfig)) {
+      return false;
+    }
+    DimensionRangePartitionsSpec partitionsSpec = (DimensionRangePartitionsSpec) tuningConfig.getPartitionsSpec();
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(
+                             dim -> ColumnType.STRING.equals(dim.getColumnType())
+                                    && partitionsSpec.getPartitionDimensions().contains(dim.getName())
+                         );
+  }
+
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  public static boolean needMultiValuedDimensions(CompactionTask compactionTask)

Review Comment:
   If only needed in test, keep the method as package private rather than public.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1804546644


##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:
##########
@@ -452,6 +454,66 @@ public boolean isPerfectRollup()
     return tuningConfig != null && tuningConfig.isForceGuaranteedRollup();
   }
 
+  /**
+   * Returns true even when granularitySpec is not known since rollup is then decided based on segment analysis.
+   */
+  static boolean isPossiblyRollup(@Nullable ClientCompactionTaskGranularitySpec granularitySpec)
+  {
+    return !(granularitySpec != null && Boolean.FALSE.equals(granularitySpec.isRollup()));
+  }
+
+  static boolean isRangePartitioned(@Nullable TuningConfig tuningConfig)
+  {
+    return tuningConfig != null && tuningConfig.getPartitionsSpec() instanceof DimensionRangePartitionsSpec;
+  }
+
+  static boolean isRollupOnStringDimension(
+      DimensionsSpec dimensionsSpec,
+      @Nullable ClientCompactionTaskGranularitySpec granularitySpec
+  )
+  {
+    if (!isPossiblyRollup(granularitySpec)) {
+      return false;
+    }
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(dim -> ColumnType.STRING.equals(dim.getColumnType()));
+  }
+
+  static boolean isPartitionedOnStringDimension(DimensionsSpec dimensionsSpec, @Nullable TuningConfig tuningConfig)
+  {
+    if (!isRangePartitioned(tuningConfig)) {
+      return false;
+    }
+    DimensionRangePartitionsSpec partitionsSpec = (DimensionRangePartitionsSpec) tuningConfig.getPartitionsSpec();
+    return dimensionsSpec.getDimensions()
+                         .stream()
+                         .anyMatch(
+                             dim -> ColumnType.STRING.equals(dim.getColumnType())
+                                    && partitionsSpec.getPartitionDimensions().contains(dim.getName())
+                         );
+  }
+
+  /**
+   * In MSQ, MVDs need conversion to array for rollup, and they aren't supported as partition columns. This check
+   * therefore is particularly for MSQCompactionRunner; it returns true when dimension types aren't known
+   * from CompactionTask spec, or if either rollup or partition dimensions contain any string-type column.
+   */
+  public static boolean needMultiValuedDimensions(CompactionTask compactionTask)

Review Comment:
   > No point making this method static if it requires passing the CompactionTask into it.
   
   Agree. One possibility is that its value can be assigned to a final field and the getter is exposed
   
   > Also, instead of making this method public or package private, CompactionTask should instead expose a method needsToDownloadSegments() or something. That can internally call this one.
   
   This would need some refactoring since the effective granularitySpec on which download depends is computed inside `createDataSchemaForIntervals`, just before passing this computed value to `ExistingSegmentAnalyzer` where the final need to download is evaluated. Also, each `need*` flag is separately communicated to the Analyzer class to do selective processing. 
   
   Does the proposed approach of assigning `needMultiValuedDimensions` to a private field and creating tests on it work? In fact, all such fields apart from `needGranularitySpec` can be populated in CompactionTask constructor and passed to the Analyzer



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Fail MSQ compaction if multi-valued partition dimensions are found (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on PR #17344:
URL: https://github.com/apache/druid/pull/17344#issuecomment-2416074099

   @kfaraz: I have now updated the logic to download segments when we need MVD info. The check is also updated to be more fine grained to avoid redundant download.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Add multi-valued check for partition dimensions (druid)

Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #17344:
URL: https://github.com/apache/druid/pull/17344#discussion_r1800617082


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java:
##########
@@ -153,11 +153,15 @@ public CompactionConfigValidationResult validateCompactionTask(
       );
     }
     List<CompactionConfigValidationResult> validationResults = new ArrayList<>();
+    DataSchema dataSchema = Iterables.getOnlyElement(intervalToDataSchemaMap.values());

Review Comment:
   The comment was added here: https://github.com/apache/druid/blob/c2149d59a7da1ac4cbb02aaa63a52cb326306dc5/extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java#L147



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org