You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2025/07/17 08:16:32 UTC

[PR] validate projection granularity is finer than or equal to segment granularity (druid)

clintropolis opened a new pull request, #18273:
URL: https://github.com/apache/druid/pull/18273

   ### Description
   Adds validation to ensure that the granularity of a projection "__time equivalent" virtual column is constrained to be finer than or equal to the segment granularity. Nothing good happens if this is allowed since the __time values of the projection no longer live within the time interval of the segment, so it is now forbidden.
   
   Also cleaned up some other projection validation messages, and cleaned up some tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2212680744


##########
server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java:
##########
@@ -131,13 +134,32 @@ public DataSchema(
     // this validation is not necessarily going to be able to catch everything. It will run again in getDimensionsSpec.
     computeAndValidateOutputFieldNames(this.dimensionsSpec, this.aggregators);
 
+
     if (this.granularitySpec.isRollup() && this.aggregators.length == 0) {
       log.warn(
           "Rollup is enabled for dataSource [%s] but no metricsSpec has been provided. "
           + "Are you sure this is what you want?",
           dataSource
       );
     }
+
+    if (this.projections != null) {
+      for (AggregateProjectionSpec projection : this.projections) {
+        final AggregateProjectionMetadata.Schema schema = projection.toMetadataSchema();
+        if (schema.getTimeColumnName() == null) {
+          continue;
+        }
+        final Granularity projectionGranularity = schema.getGranularity();
+        if (this.granularitySpec.getSegmentGranularity().isFinerThan(projectionGranularity)) {
+          throw InvalidInput.exception(
+              "Projection[%s] has granularity[%s] must be finer than or equal to segment granularity[%s]",

Review Comment:
   Reads a little odd:
   
   ```suggestion
                 "Projection[%s] has granularity[%s] which is coarser than segment granularity[%s]",
   ```



##########
server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java:
##########
@@ -131,13 +134,32 @@ public DataSchema(
     // this validation is not necessarily going to be able to catch everything. It will run again in getDimensionsSpec.
     computeAndValidateOutputFieldNames(this.dimensionsSpec, this.aggregators);
 
+

Review Comment:
   Nit: extra line.



##########
server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java:
##########
@@ -131,13 +134,32 @@ public DataSchema(
     // this validation is not necessarily going to be able to catch everything. It will run again in getDimensionsSpec.
     computeAndValidateOutputFieldNames(this.dimensionsSpec, this.aggregators);
 
+
     if (this.granularitySpec.isRollup() && this.aggregators.length == 0) {
       log.warn(
           "Rollup is enabled for dataSource [%s] but no metricsSpec has been provided. "
           + "Are you sure this is what you want?",
           dataSource
       );
     }
+
+    if (this.projections != null) {

Review Comment:
   nit: may be this part can be put in a separate method?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2214256092


##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/exec/MSQInsertTest.java:
##########
@@ -614,6 +655,77 @@
 
   }
 
+  @MethodSource("data")
+  @ParameterizedTest(name = "{index}:with context {0}")
+  public void testInsertOnExternalDataSourceWithCatalogProjectionsInvalidGranularity(
+      String contextName,
+      Map<String, Object> context
+  ) throws IOException
+  {
+    final File toRead = getResourceAsTemporaryFile("/wikipedia-sampled.json");
+    final String toReadFileNameAsJson = queryFramework().queryJsonMapper().writeValueAsString(toRead.getAbsolutePath());
+
+
+    testIngestQuery().setSql(" insert into foo2 SELECT\n"
+                             + "  floor(TIME_PARSE(\"timestamp\") to minute) AS __time,\n"
+                             + "  channel,\n"
+                             + "  page,\n"
+                             + "  user,\n"
+                             + "  added,\n"
+                             + "  deleted,\n"
+                             + "  delta\n"
+                             + "FROM TABLE(\n"
+                             + "  EXTERN(\n"
+                             + "    '{ \"files\": [" + toReadFileNameAsJson + "],\"type\":\"local\"}',\n"
+                             + "    '{\"type\": \"json\"}',\n"
+                             + "    '[{\"name\": \"timestamp\", \"type\": \"string\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"page\", \"type\": \"string\"}, {\"name\": \"user\", \"type\": \"string\"}, {\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"deleted\", \"type\": \"long\"}, {\"name\": \"delta\", \"type\": \"long\"}]'\n"
+                             + "  )\n"
+                             + ") PARTITIONED by hour")
+                     .setExpectedValidationErrorMatcher(
+                         DruidExceptionMatcher.invalidInput().expectMessageIs(
+                             "projection[channel_delta_daily] has granularity[{type=period, period=P1D, timeZone=UTC, origin=null}] which must be finer than or equal to segment granularity[{type=period, period=PT1H, timeZone=UTC, origin=null}]"
+                         )
+                     )
+                     .verifyPlanningErrors();
+
+  }
+
+  @MethodSource("data")
+  @ParameterizedTest(name = "{index}:with context {0}")
+  public void testInsertOnExternalDataSourceWithCatalogProjectionsMissingColumn(
+      String contextName,
+      Map<String, Object> context

Review Comment:
   ## Useless parameter
   
   The parameter 'context' is never used.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9914)



##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/exec/MSQInsertTest.java:
##########
@@ -614,6 +655,77 @@
 
   }
 
+  @MethodSource("data")
+  @ParameterizedTest(name = "{index}:with context {0}")
+  public void testInsertOnExternalDataSourceWithCatalogProjectionsInvalidGranularity(
+      String contextName,

Review Comment:
   ## Useless parameter
   
   The parameter 'contextName' is never used.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9915)



##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/exec/MSQInsertTest.java:
##########
@@ -614,6 +655,77 @@
 
   }
 
+  @MethodSource("data")
+  @ParameterizedTest(name = "{index}:with context {0}")
+  public void testInsertOnExternalDataSourceWithCatalogProjectionsInvalidGranularity(
+      String contextName,
+      Map<String, Object> context
+  ) throws IOException
+  {
+    final File toRead = getResourceAsTemporaryFile("/wikipedia-sampled.json");
+    final String toReadFileNameAsJson = queryFramework().queryJsonMapper().writeValueAsString(toRead.getAbsolutePath());
+
+
+    testIngestQuery().setSql(" insert into foo2 SELECT\n"
+                             + "  floor(TIME_PARSE(\"timestamp\") to minute) AS __time,\n"
+                             + "  channel,\n"
+                             + "  page,\n"
+                             + "  user,\n"
+                             + "  added,\n"
+                             + "  deleted,\n"
+                             + "  delta\n"
+                             + "FROM TABLE(\n"
+                             + "  EXTERN(\n"
+                             + "    '{ \"files\": [" + toReadFileNameAsJson + "],\"type\":\"local\"}',\n"
+                             + "    '{\"type\": \"json\"}',\n"
+                             + "    '[{\"name\": \"timestamp\", \"type\": \"string\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"page\", \"type\": \"string\"}, {\"name\": \"user\", \"type\": \"string\"}, {\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"deleted\", \"type\": \"long\"}, {\"name\": \"delta\", \"type\": \"long\"}]'\n"
+                             + "  )\n"
+                             + ") PARTITIONED by hour")
+                     .setExpectedValidationErrorMatcher(
+                         DruidExceptionMatcher.invalidInput().expectMessageIs(
+                             "projection[channel_delta_daily] has granularity[{type=period, period=P1D, timeZone=UTC, origin=null}] which must be finer than or equal to segment granularity[{type=period, period=PT1H, timeZone=UTC, origin=null}]"
+                         )
+                     )
+                     .verifyPlanningErrors();
+
+  }
+
+  @MethodSource("data")
+  @ParameterizedTest(name = "{index}:with context {0}")
+  public void testInsertOnExternalDataSourceWithCatalogProjectionsMissingColumn(
+      String contextName,

Review Comment:
   ## Useless parameter
   
   The parameter 'contextName' is never used.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9913)



##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/exec/MSQInsertTest.java:
##########
@@ -614,6 +655,77 @@
 
   }
 
+  @MethodSource("data")
+  @ParameterizedTest(name = "{index}:with context {0}")
+  public void testInsertOnExternalDataSourceWithCatalogProjectionsInvalidGranularity(
+      String contextName,
+      Map<String, Object> context

Review Comment:
   ## Useless parameter
   
   The parameter 'context' is never used.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9916)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2212734683


##########
server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java:
##########
@@ -131,13 +134,32 @@ public DataSchema(
     // this validation is not necessarily going to be able to catch everything. It will run again in getDimensionsSpec.
     computeAndValidateOutputFieldNames(this.dimensionsSpec, this.aggregators);
 
+
     if (this.granularitySpec.isRollup() && this.aggregators.length == 0) {
       log.warn(
           "Rollup is enabled for dataSource [%s] but no metricsSpec has been provided. "
           + "Are you sure this is what you want?",
           dataSource
       );
     }
+
+    if (this.projections != null) {
+      for (AggregateProjectionSpec projection : this.projections) {
+        final AggregateProjectionMetadata.Schema schema = projection.toMetadataSchema();
+        if (schema.getTimeColumnName() == null) {
+          continue;
+        }
+        final Granularity projectionGranularity = schema.getGranularity();
+        if (this.granularitySpec.getSegmentGranularity().isFinerThan(projectionGranularity)) {
+          throw InvalidInput.exception(
+              "Projection[%s] has granularity[%s] must be finer than or equal to segment granularity[%s]",

Review Comment:
   Yeah, that works too!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2212659849


##########
processing/src/main/java/org/apache/druid/segment/AggregateProjectionMetadata.java:
##########
@@ -182,9 +182,15 @@ public Schema(
         @JsonProperty("ordering") List<OrderBy> ordering
     )
     {
+      if (name == null) {
+        throw DruidException.defensive("projection schema name cannot be null");

Review Comment:
   Should these be defensive or invalid input?



##########
processing/src/test/java/org/apache/druid/segment/AggregateProjectionMetadataTest.java:
##########
@@ -30,19 +30,19 @@
 import org.apache.druid.query.aggregation.CountAggregatorFactory;
 import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
 import org.apache.druid.testing.InitializedNullHandlingTest;
-import org.junit.Assert;
-import org.junit.Test;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Assertions;
 
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.SortedSet;
 
-public class AggregateProjectionMetadataTest extends InitializedNullHandlingTest
+class AggregateProjectionMetadataTest extends InitializedNullHandlingTest
 {
   private static final ObjectMapper JSON_MAPPER = TestHelper.makeJsonMapper();
 
   @Test
-  public void testSerde() throws JsonProcessingException
+  void testSerde() throws JsonProcessingException

Review Comment:
   Just curious, any specific reason for removing the `public` from the class and methods?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "cecemei (via GitHub)" <gi...@apache.org>.
cecemei commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2213840241


##########
processing/src/main/java/org/apache/druid/data/input/impl/AggregateProjectionSpec.java:
##########
@@ -73,9 +73,12 @@ public AggregateProjectionSpec(
       @JsonProperty("aggregators") @Nullable AggregatorFactory[] aggregators
   )
   {
+    if (name == null) {

Review Comment:
   nit: maybe `StringUtils.isEmpty` to prevent name being empty string as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2212809882


##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -441,29 +454,32 @@
         ), JacksonUtils.TYPE_REFERENCE_MAP_STRING_OBJECT
     );
 
-    expectedException.expect(DruidException.class);
-    expectedException.expectMessage(
-        "Cannot specify a column more than once: [metric1] seen in metricsSpec list (2 occurrences); "
-        + "[metric3] seen in metricsSpec list (2 occurrences)"
+    Throwable t = Assertions.assertThrows(
+        DruidException.class,
+        () -> DataSchema.builder()
+                        .withDataSource(IdUtilsTest.VALID_ID_CHARS)
+                        .withParserMap(parser)
+                        .withAggregators(
+                            new DoubleSumAggregatorFactory("metric1", "col1"),
+                            new DoubleSumAggregatorFactory("metric2", "col2"),
+                            new DoubleSumAggregatorFactory("metric1", "col3"),
+                            new DoubleSumAggregatorFactory("metric3", "col4"),
+                            new DoubleSumAggregatorFactory("metric3", "col5")
+                        )
+                        .withGranularity(ARBITRARY_GRANULARITY)
+                        .withObjectMapper(jsonMapper)

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [Builder.withObjectMapper](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9905)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -284,72 +284,81 @@
                                   .withObjectMapper(jsonMapper)
                                   .build();
 
-    Assert.assertEquals(
+    Assertions.assertEquals(
         ImmutableList.of("__time", "dimA", "dimB"),
         schema.getDimensionsSpec().getDimensionNames()
     );
 
-    Assert.assertTrue(schema.getDimensionsSpec().isForceSegmentSortByTime());
+    Assertions.assertTrue(schema.getDimensionsSpec().isForceSegmentSortByTime());
   }
 
   @Test
-  public void testOverlapTimeAndDimPositionZeroWrongType()
+  void testOverlapTimeAndDimPositionZeroWrongType()
   {
-    expectedException.expect(DruidException.class);
-    expectedException.expectMessage("Encountered dimension[__time] with incorrect type[STRING]. Type must be 'long'.");
-
-    DataSchema.builder()
-              .withDataSource(IdUtilsTest.VALID_ID_CHARS)
-              .withTimestamp(new TimestampSpec("time", "auto", null))
-              .withDimensions(
-                  DimensionsSpec.builder()
-                                .setDimensions(
-                                    ImmutableList.of(
-                                        new StringDimensionSchema("__time"),
-                                        new StringDimensionSchema("dimA"),
-                                        new StringDimensionSchema("dimB")
-                                    )
-                                )
-                                .setDimensionExclusions(ImmutableList.of("dimC"))
-                                .build()
-              )
-              .withGranularity(ARBITRARY_GRANULARITY)
-              .withObjectMapper(jsonMapper)
-              .build();
+    Throwable t = Assertions.assertThrows(
+        DruidException.class,
+        () -> DataSchema.builder()
+                        .withDataSource(IdUtilsTest.VALID_ID_CHARS)
+                        .withTimestamp(new TimestampSpec("time", "auto", null))
+                        .withDimensions(
+                            DimensionsSpec.builder()
+                                          .setDimensions(
+                                              ImmutableList.of(
+                                                  new StringDimensionSchema("__time"),
+                                                  new StringDimensionSchema("dimA"),
+                                                  new StringDimensionSchema("dimB")
+                                              )
+                                          )
+                                          .setDimensionExclusions(ImmutableList.of("dimC"))
+                                          .build()
+                        )
+                        .withGranularity(ARBITRARY_GRANULARITY)
+                        .withObjectMapper(jsonMapper)
+                        .build()
+    );
+
+    Assertions.assertEquals(
+        "Encountered dimension[__time] with incorrect type[STRING]. Type must be 'long'.",
+        t.getMessage()
+    );
   }
 
   @Test
-  public void testOverlapTimeAndDimPositionOne()
+  void testOverlapTimeAndDimPositionOne()
   {
-    expectedException.expect(DruidException.class);
-    expectedException.expectMessage(
+
+    Throwable t = Assertions.assertThrows(
+        DruidException.class,
+        () -> DataSchema.builder()
+                        .withDataSource(IdUtilsTest.VALID_ID_CHARS)
+                        .withTimestamp(new TimestampSpec("time", "auto", null))
+                        .withDimensions(
+                            DimensionsSpec.builder()
+                                          .setDimensions(
+                                              ImmutableList.of(
+                                                  new StringDimensionSchema("dimA"),
+                                                  new LongDimensionSchema("__time"),
+                                                  new StringDimensionSchema("dimB")
+                                              )
+                                          )
+                                          .setDimensionExclusions(ImmutableList.of("dimC"))
+                                          .build()
+                        )
+                        .withGranularity(ARBITRARY_GRANULARITY)
+                        .withObjectMapper(jsonMapper)

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [Builder.withObjectMapper](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9904)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -774,16 +794,16 @@
     String serialized = jsonMapper.writeValueAsString(originalSchema);
     DataSchema deserialized = jsonMapper.readValue(serialized, DataSchema.class);
 
-    Assert.assertEquals(originalSchema.getDataSource(), deserialized.getDataSource());
-    Assert.assertEquals(originalSchema.getGranularitySpec(), deserialized.getGranularitySpec());
-    Assert.assertEquals(originalSchema.getParser().getParseSpec(), deserialized.getParser().getParseSpec());
-    Assert.assertArrayEquals(originalSchema.getAggregators(), deserialized.getAggregators());
-    Assert.assertEquals(originalSchema.getTransformSpec(), deserialized.getTransformSpec());
-    Assert.assertEquals(originalSchema.getParserMap(), deserialized.getParserMap());
+    Assertions.assertEquals(originalSchema.getDataSource(), deserialized.getDataSource());
+    Assertions.assertEquals(originalSchema.getGranularitySpec(), deserialized.getGranularitySpec());
+    Assertions.assertEquals(originalSchema.getParser().getParseSpec(), deserialized.getParser().getParseSpec());
+    Assertions.assertArrayEquals(originalSchema.getAggregators(), deserialized.getAggregators());
+    Assertions.assertEquals(originalSchema.getTransformSpec(), deserialized.getTransformSpec());
+    Assertions.assertEquals(originalSchema.getParserMap(), deserialized.getParserMap());

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [DataSchema.getParserMap](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9910)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -806,18 +826,18 @@
                                      .withObjectMapper(jsonMapper)
                                      .build();
     DataSchema newSchema = oldSchema.withDimensionsSpec(newDimSpec);
-    Assert.assertSame(oldSchema.getDataSource(), newSchema.getDataSource());
-    Assert.assertSame(oldSchema.getTimestampSpec(), newSchema.getTimestampSpec());
-    Assert.assertSame(newDimSpec, newSchema.getDimensionsSpec());
-    Assert.assertSame(oldSchema.getAggregators(), newSchema.getAggregators());
-    Assert.assertSame(oldSchema.getGranularitySpec(), newSchema.getGranularitySpec());
-    Assert.assertSame(oldSchema.getTransformSpec(), newSchema.getTransformSpec());
-    Assert.assertSame(oldSchema.getParserMap(), newSchema.getParserMap());
+    Assertions.assertSame(oldSchema.getDataSource(), newSchema.getDataSource());
+    Assertions.assertSame(oldSchema.getTimestampSpec(), newSchema.getTimestampSpec());
+    Assertions.assertSame(newDimSpec, newSchema.getDimensionsSpec());
+    Assertions.assertSame(oldSchema.getAggregators(), newSchema.getAggregators());
+    Assertions.assertSame(oldSchema.getGranularitySpec(), newSchema.getGranularitySpec());
+    Assertions.assertSame(oldSchema.getTransformSpec(), newSchema.getTransformSpec());
+    Assertions.assertSame(oldSchema.getParserMap(), newSchema.getParserMap());

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [DataSchema.getParserMap](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9911)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -806,18 +826,18 @@
                                      .withObjectMapper(jsonMapper)
                                      .build();
     DataSchema newSchema = oldSchema.withDimensionsSpec(newDimSpec);
-    Assert.assertSame(oldSchema.getDataSource(), newSchema.getDataSource());
-    Assert.assertSame(oldSchema.getTimestampSpec(), newSchema.getTimestampSpec());
-    Assert.assertSame(newDimSpec, newSchema.getDimensionsSpec());
-    Assert.assertSame(oldSchema.getAggregators(), newSchema.getAggregators());
-    Assert.assertSame(oldSchema.getGranularitySpec(), newSchema.getGranularitySpec());
-    Assert.assertSame(oldSchema.getTransformSpec(), newSchema.getTransformSpec());
-    Assert.assertSame(oldSchema.getParserMap(), newSchema.getParserMap());
+    Assertions.assertSame(oldSchema.getDataSource(), newSchema.getDataSource());
+    Assertions.assertSame(oldSchema.getTimestampSpec(), newSchema.getTimestampSpec());
+    Assertions.assertSame(newDimSpec, newSchema.getDimensionsSpec());
+    Assertions.assertSame(oldSchema.getAggregators(), newSchema.getAggregators());
+    Assertions.assertSame(oldSchema.getGranularitySpec(), newSchema.getGranularitySpec());
+    Assertions.assertSame(oldSchema.getTransformSpec(), newSchema.getTransformSpec());
+    Assertions.assertSame(oldSchema.getParserMap(), newSchema.getParserMap());

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [DataSchema.getParserMap](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9912)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -731,17 +751,17 @@
     String serialized = jsonMapper.writeValueAsString(originalSchema);
     TestModifiedDataSchema deserialized = jsonMapper.readValue(serialized, TestModifiedDataSchema.class);
 
-    Assert.assertEquals(null, deserialized.getExtra());
-    Assert.assertEquals(originalSchema.getDataSource(), deserialized.getDataSource());
-    Assert.assertEquals(originalSchema.getGranularitySpec(), deserialized.getGranularitySpec());
-    Assert.assertEquals(originalSchema.getParser().getParseSpec(), deserialized.getParser().getParseSpec());
-    Assert.assertArrayEquals(originalSchema.getAggregators(), deserialized.getAggregators());
-    Assert.assertEquals(originalSchema.getTransformSpec(), deserialized.getTransformSpec());
-    Assert.assertEquals(originalSchema.getParserMap(), deserialized.getParserMap());
+    Assertions.assertEquals(null, deserialized.getExtra());
+    Assertions.assertEquals(originalSchema.getDataSource(), deserialized.getDataSource());
+    Assertions.assertEquals(originalSchema.getGranularitySpec(), deserialized.getGranularitySpec());
+    Assertions.assertEquals(originalSchema.getParser().getParseSpec(), deserialized.getParser().getParseSpec());
+    Assertions.assertArrayEquals(originalSchema.getAggregators(), deserialized.getAggregators());
+    Assertions.assertEquals(originalSchema.getTransformSpec(), deserialized.getTransformSpec());
+    Assertions.assertEquals(originalSchema.getParserMap(), deserialized.getParserMap());

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [DataSchema.getParserMap](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9908)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -284,72 +284,81 @@
                                   .withObjectMapper(jsonMapper)
                                   .build();
 
-    Assert.assertEquals(
+    Assertions.assertEquals(
         ImmutableList.of("__time", "dimA", "dimB"),
         schema.getDimensionsSpec().getDimensionNames()
     );
 
-    Assert.assertTrue(schema.getDimensionsSpec().isForceSegmentSortByTime());
+    Assertions.assertTrue(schema.getDimensionsSpec().isForceSegmentSortByTime());
   }
 
   @Test
-  public void testOverlapTimeAndDimPositionZeroWrongType()
+  void testOverlapTimeAndDimPositionZeroWrongType()
   {
-    expectedException.expect(DruidException.class);
-    expectedException.expectMessage("Encountered dimension[__time] with incorrect type[STRING]. Type must be 'long'.");
-
-    DataSchema.builder()
-              .withDataSource(IdUtilsTest.VALID_ID_CHARS)
-              .withTimestamp(new TimestampSpec("time", "auto", null))
-              .withDimensions(
-                  DimensionsSpec.builder()
-                                .setDimensions(
-                                    ImmutableList.of(
-                                        new StringDimensionSchema("__time"),
-                                        new StringDimensionSchema("dimA"),
-                                        new StringDimensionSchema("dimB")
-                                    )
-                                )
-                                .setDimensionExclusions(ImmutableList.of("dimC"))
-                                .build()
-              )
-              .withGranularity(ARBITRARY_GRANULARITY)
-              .withObjectMapper(jsonMapper)
-              .build();
+    Throwable t = Assertions.assertThrows(
+        DruidException.class,
+        () -> DataSchema.builder()
+                        .withDataSource(IdUtilsTest.VALID_ID_CHARS)
+                        .withTimestamp(new TimestampSpec("time", "auto", null))
+                        .withDimensions(
+                            DimensionsSpec.builder()
+                                          .setDimensions(
+                                              ImmutableList.of(
+                                                  new StringDimensionSchema("__time"),
+                                                  new StringDimensionSchema("dimA"),
+                                                  new StringDimensionSchema("dimB")
+                                              )
+                                          )
+                                          .setDimensionExclusions(ImmutableList.of("dimC"))
+                                          .build()
+                        )
+                        .withGranularity(ARBITRARY_GRANULARITY)
+                        .withObjectMapper(jsonMapper)

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [Builder.withObjectMapper](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9903)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -441,29 +454,32 @@
         ), JacksonUtils.TYPE_REFERENCE_MAP_STRING_OBJECT
     );
 
-    expectedException.expect(DruidException.class);
-    expectedException.expectMessage(
-        "Cannot specify a column more than once: [metric1] seen in metricsSpec list (2 occurrences); "
-        + "[metric3] seen in metricsSpec list (2 occurrences)"
+    Throwable t = Assertions.assertThrows(
+        DruidException.class,
+        () -> DataSchema.builder()
+                        .withDataSource(IdUtilsTest.VALID_ID_CHARS)
+                        .withParserMap(parser)

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [Builder.withParserMap](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9906)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -731,17 +751,17 @@
     String serialized = jsonMapper.writeValueAsString(originalSchema);
     TestModifiedDataSchema deserialized = jsonMapper.readValue(serialized, TestModifiedDataSchema.class);
 
-    Assert.assertEquals(null, deserialized.getExtra());
-    Assert.assertEquals(originalSchema.getDataSource(), deserialized.getDataSource());
-    Assert.assertEquals(originalSchema.getGranularitySpec(), deserialized.getGranularitySpec());
-    Assert.assertEquals(originalSchema.getParser().getParseSpec(), deserialized.getParser().getParseSpec());
-    Assert.assertArrayEquals(originalSchema.getAggregators(), deserialized.getAggregators());
-    Assert.assertEquals(originalSchema.getTransformSpec(), deserialized.getTransformSpec());
-    Assert.assertEquals(originalSchema.getParserMap(), deserialized.getParserMap());
+    Assertions.assertEquals(null, deserialized.getExtra());
+    Assertions.assertEquals(originalSchema.getDataSource(), deserialized.getDataSource());
+    Assertions.assertEquals(originalSchema.getGranularitySpec(), deserialized.getGranularitySpec());
+    Assertions.assertEquals(originalSchema.getParser().getParseSpec(), deserialized.getParser().getParseSpec());
+    Assertions.assertArrayEquals(originalSchema.getAggregators(), deserialized.getAggregators());
+    Assertions.assertEquals(originalSchema.getTransformSpec(), deserialized.getTransformSpec());
+    Assertions.assertEquals(originalSchema.getParserMap(), deserialized.getParserMap());

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [DataSchema.getParserMap](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9907)



##########
server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java:
##########
@@ -774,16 +794,16 @@
     String serialized = jsonMapper.writeValueAsString(originalSchema);
     DataSchema deserialized = jsonMapper.readValue(serialized, DataSchema.class);
 
-    Assert.assertEquals(originalSchema.getDataSource(), deserialized.getDataSource());
-    Assert.assertEquals(originalSchema.getGranularitySpec(), deserialized.getGranularitySpec());
-    Assert.assertEquals(originalSchema.getParser().getParseSpec(), deserialized.getParser().getParseSpec());
-    Assert.assertArrayEquals(originalSchema.getAggregators(), deserialized.getAggregators());
-    Assert.assertEquals(originalSchema.getTransformSpec(), deserialized.getTransformSpec());
-    Assert.assertEquals(originalSchema.getParserMap(), deserialized.getParserMap());
+    Assertions.assertEquals(originalSchema.getDataSource(), deserialized.getDataSource());
+    Assertions.assertEquals(originalSchema.getGranularitySpec(), deserialized.getGranularitySpec());
+    Assertions.assertEquals(originalSchema.getParser().getParseSpec(), deserialized.getParser().getParseSpec());
+    Assertions.assertArrayEquals(originalSchema.getAggregators(), deserialized.getAggregators());
+    Assertions.assertEquals(originalSchema.getTransformSpec(), deserialized.getTransformSpec());
+    Assertions.assertEquals(originalSchema.getParserMap(), deserialized.getParserMap());

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [DataSchema.getParserMap](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/9909)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #18273:
URL: https://github.com/apache/druid/pull/18273#issuecomment-3088779924

   CI failure seems unrelated to this PR (seeing it in other places too)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2212686426


##########
processing/src/test/java/org/apache/druid/segment/AggregateProjectionMetadataTest.java:
##########
@@ -30,19 +30,19 @@
 import org.apache.druid.query.aggregation.CountAggregatorFactory;
 import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
 import org.apache.druid.testing.InitializedNullHandlingTest;
-import org.junit.Assert;
-import org.junit.Test;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Assertions;
 
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.SortedSet;
 
-public class AggregateProjectionMetadataTest extends InitializedNullHandlingTest
+class AggregateProjectionMetadataTest extends InitializedNullHandlingTest
 {
   private static final ObjectMapper JSON_MAPPER = TestHelper.makeJsonMapper();
 
   @Test
-  public void testSerde() throws JsonProcessingException
+  void testSerde() throws JsonProcessingException

Review Comment:
   not really, just they also aren't needed so just went ahead and did it while changing to use junit5 stuff :shrug:



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2212675475


##########
processing/src/main/java/org/apache/druid/segment/AggregateProjectionMetadata.java:
##########
@@ -182,9 +182,15 @@ public Schema(
         @JsonProperty("ordering") List<OrderBy> ordering
     )
     {
+      if (name == null) {
+        throw DruidException.defensive("projection schema name cannot be null");

Review Comment:
   these are defensive because this object is created internally from `AggregateProjectionSpec` which is the user facing object that does invalid input



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis merged PR #18273:
URL: https://github.com/apache/druid/pull/18273


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] validate projection granularity is finer than or equal to segment granularity (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #18273:
URL: https://github.com/apache/druid/pull/18273#discussion_r2212691633


##########
server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java:
##########
@@ -131,13 +134,32 @@ public DataSchema(
     // this validation is not necessarily going to be able to catch everything. It will run again in getDimensionsSpec.
     computeAndValidateOutputFieldNames(this.dimensionsSpec, this.aggregators);
 
+
     if (this.granularitySpec.isRollup() && this.aggregators.length == 0) {
       log.warn(
           "Rollup is enabled for dataSource [%s] but no metricsSpec has been provided. "
           + "Are you sure this is what you want?",
           dataSource
       );
     }
+
+    if (this.projections != null) {
+      for (AggregateProjectionSpec projection : this.projections) {
+        final AggregateProjectionMetadata.Schema schema = projection.toMetadataSchema();
+        if (schema.getTimeColumnName() == null) {
+          continue;
+        }
+        final Granularity projectionGranularity = schema.getGranularity();
+        if (this.granularitySpec.getSegmentGranularity().isFinerThan(projectionGranularity)) {
+          throw InvalidInput.exception(
+              "Projection[%s] has granularity[%s] must be finer than or equal to segment granularity[%s]",

Review Comment:
   oops, i agree it needs the "which:, but i think "finer than or equal to" makes it more obvious what you need to do as a user since that is the change you need to make to the projection granularity



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org