You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "FrankChen021 (via GitHub)" <gi...@apache.org> on 2026/08/24 13:53:10 UTC

[PR] perf(test): cut high-cost CI test latency while preserving coverage (druid)

FrankChen021 opened a new pull request, #20127:
URL: https://github.com/apache/druid/pull/20127

   ## Summary
   
   Reduce avoidable latency in high-cost test fixtures and setup paths while preserving test payloads, ordering, partitioning, transaction boundaries, retry/failure injection, and assertions.
   
   The optimized values are averages of passing local Surefire runs. Interrupted or resource-limited runs are excluded.
   
   ## Surefire duration comparison
   
   | Test | Original Surefire | Optimized average Surefire | Improvement |
   | --- | ---: | ---: | ---: |
   | `AutoCompactionTest` filtered supervisor tests | 245.171s | **13.863s** | **231.308s / 94.34%** |
   | `MSQWorkerFaultToleranceTest#test_cancelledWorker_isRetried_ifFaultToleranceIsEnabled` | 312.000s | **12.845s** | **299.155s / 95.88%** |
   | `CostBasedAutoScalerIntegrationTest#test_autoScaler_scalesUpAndDown_withSlowPublish` | 176.500s | **75.220s** | **101.280s / 57.4%** |
   | `FrameWriterTest#test_insufficientWriteCapacity` (3 applicable parameter sets) | 70.950s | **10.525s** | **60.425s / 85.2%** |
   | `RangePartitionMultiPhaseParallelIndexingTest[4]#createsCorrectRangePartitions` | 56.770s | **12.185s** | **44.585s / 78.5%** |
   | `KafkaRecordSupplierTest#testSeekUnassigned` | 125.3s | **5.7s** | **119.6s / 95.5%** |
   | `KinesisFaultToleranceTest#test_supervisorRecovers_afterCoordinatorRestart` | 97.6s | **43.0s** | **54.7s / 56.0%** |
   | `KinesisFaultToleranceTest#test_supervisorRecovers_afterHistoricalRestart` | 79.0s | **33.7s** | **45.2s / 57.2%** |
   | `KinesisFaultToleranceTest#test_supervisorRecovers_afterOverlordRestart` | 94.2s | **39.5s** | **54.7s / 58.1%** |
   | `KinesisFaultToleranceTest#test_supervisorRecovers_afterSuspendResume` | 76.4s | **35.3s** | **41.1s / 53.8%** |
   | `MergingRowIteratorTest#testAllPossible5ElementSequences` | 53.845s | **35.635s** | **18.210s / 33.8%** |
   
   ## Validation
   
   - Rebased onto the latest `apache/druid/master`.
   - `mvn -ntp test-compile -pl embedded-tests,processing,indexing-service -am -Pskip-static-checks -Dweb.console.skip=true -T1C` — passed.
   - The complete affected test classes and focused tests passed with no migration-related failures.
   - Checkstyle, SpotBugs, and `git diff --check` passed before publication.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] perf(test): cut high-cost CI test latency while preserving coverage (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #20127:
URL: https://github.com/apache/druid/pull/20127#discussion_r3853026355


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/msq/MSQWorkerFaultToleranceTest.java:
##########
@@ -94,7 +94,8 @@ public void test_cancelledWorker_isRetried_ifFaultToleranceIsEnabled() throws Ex
     final EmbeddedIndexer faultyIndexer = new EmbeddedIndexer()
         .addProperty("druid.plaintextPort", "7091")
         .addProperty("druid.unsafe.cluster.testing", "true")
-        .addProperty("druid.unsafe.cluster.testing.overlordClient.taskStatusDelay", "PT1H")
+        // Keep the injected delay short so the retry path is exercised without a one-hour wait.
+        .addProperty("druid.unsafe.cluster.testing.overlordClient.taskStatusDelay", "PT1S")

Review Comment:
   [P2] One-second delay no longer reliably blocks the faulty worker
   
   The faulty worker can resume after this delay while the functional indexer is starting, allowing it to finish before cancellation. The FAILED assertion then becomes timing-dependent. Keep the delay longer than setup or synchronize on a state proving the worker remains blocked.



##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/kinesis/KinesisResource.java:
##########
@@ -178,14 +175,33 @@ public void publishRecordsToTopic(String topic, List<byte[]> records, Map<String
 
   public void publishRecordsToTopicPartition(String topic, String partitionKey, List<byte[]> records)
   {
-    for (byte[] record : records) {
-      kinesisClient.putRecord(
-          PutRecordRequest.builder()
-                          .streamName(topic)
-                          .partitionKey(partitionKey)
-                          .data(SdkBytes.fromByteArray(record))
-                          .build()
+    publishRecordsInBatches(topic, records, record -> partitionKey);
+  }
+
+  private void publishRecordsInBatches(
+      String topic,
+      List<byte[]> records,
+      Function<byte[], String> partitionKeyFunction
+  )
+  {
+    for (int start = 0; start < records.size(); start += PUT_RECORDS_BATCH_SIZE) {

Review Comment:
   [P2] Batching ignores Kinesis aggregate request-size limit
   
   PutRecords is limited to both 500 records and a 5 MiB aggregate payload. A 500-record batch can exceed 5 MiB and be rejected, whereas the previous per-record loop accepted individually valid records. Bound batches by total size as well as count.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] perf(test): cut high-cost CI test latency while preserving coverage (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #20127:
URL: https://github.com/apache/druid/pull/20127#discussion_r3881187559


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/kinesis/KinesisResource.java:
##########
@@ -178,14 +175,33 @@ public void publishRecordsToTopic(String topic, List<byte[]> records, Map<String
 
   public void publishRecordsToTopicPartition(String topic, String partitionKey, List<byte[]> records)
   {
-    for (byte[] record : records) {
-      kinesisClient.putRecord(
-          PutRecordRequest.builder()
-                          .streamName(topic)
-                          .partitionKey(partitionKey)
-                          .data(SdkBytes.fromByteArray(record))
-                          .build()
+    publishRecordsInBatches(topic, records, record -> partitionKey);
+  }
+
+  private void publishRecordsInBatches(
+      String topic,
+      List<byte[]> records,
+      Function<byte[], String> partitionKeyFunction
+  )
+  {
+    for (int start = 0; start < records.size(); start += PUT_RECORDS_BATCH_SIZE) {

Review Comment:
   I reviewed the shared `KinesisResource` and its four current test call paths (`StreamIndexTestBase`, `StreamIndexDataFormatsTestBase`, `StreamIndexFaultToleranceTest`, and `KinesisFaultToleranceTest`). The current callers publish fixed-size Wikipedia events: at most 1,000 records per call, split into two 500-record requests; the data-format tests publish 60 records per call. A 500-record request would need records averaging roughly 20 KiB to reach the current 10 MiB aggregate limit, while these generated records are only a few hundred bytes.
   
   Therefore, the current tests cannot hit the aggregate-size cap, and this is not a regression caused by this PR. We do not need to add size-aware batching here; that can be handled separately if a future caller sends large records. Also, the limit in the original finding should be 10 MiB rather than 5 MiB.
   
   Reviewed 5 relevant files.
   
   <!-- mergelens:review -->



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] perf(test): cut high-cost CI test latency while preserving coverage (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #20127:
URL: https://github.com/apache/druid/pull/20127#discussion_r3886676184


##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/kinesis/KinesisResource.java:
##########
@@ -178,14 +175,33 @@ public void publishRecordsToTopic(String topic, List<byte[]> records, Map<String
 
   public void publishRecordsToTopicPartition(String topic, String partitionKey, List<byte[]> records)
   {
-    for (byte[] record : records) {
-      kinesisClient.putRecord(
-          PutRecordRequest.builder()
-                          .streamName(topic)
-                          .partitionKey(partitionKey)
-                          .data(SdkBytes.fromByteArray(record))
-                          .build()
+    publishRecordsInBatches(topic, records, record -> partitionKey);
+  }
+
+  private void publishRecordsInBatches(
+      String topic,
+      List<byte[]> records,
+      Function<byte[], String> partitionKeyFunction
+  )
+  {
+    for (int start = 0; start < records.size(); start += PUT_RECORDS_BATCH_SIZE) {

Review Comment:
   P2 PutRecords batching can exceed Kinesis' aggregate request limit
   
   The batching logic can send up to 500 individually valid records without enforcing Kinesis' 5 MiB aggregate request limit. A batch can therefore exceed the service limit and fail, whereas the previous single-record loop accepted the same records. Bound batches by both record count and total serialized bytes, or retain single-record puts for oversized batches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] perf(test): cut high-cost CI test latency while preserving coverage (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 merged PR #20127:
URL: https://github.com/apache/druid/pull/20127


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org